Just saw Ari Zilka’s talk on Cluster Visualization at JavaOne. It leans heavily on Terracotta of course. It was a great presentation because it gave a brief overview on what Terracotta does and then dove into looking at a sample application and using cluster visualization to explore the performance of the app. That includes looking at transactional stats, object flush/fault rates, lock statistics, and more.
We’ve added a ton of new stuff in our most recent 2.6 release to help you understand what’s happening in the cluster, take thread dumps across the cluster with a button push, find bottlenecked locks, record performance data in a session and view the data snapshot later. When our field guys first saw the cluster thread dumping, their jaws dropped. Collecting that info is such a pain for them and customers and now it’s soooo much easier.
I tried to keep track of a bunch of questions that happened during the talk. I missed a few at the beginning but here are the ones I recorded (with answers!):
- Do changes go peer to peer? - No. Changes go to the active server (1 active server and 0+ passives for hot failover, active/active with striping in work), then are broadcast out to all nodes *that have the object*. Nodes that don’t currently have it don’t need the broadcast.
- Are clustered locks fair? - In Terracotta 2.5 and earlier locks were fair. If one client and another requested, the other client was granted. In 2.6, locks are now less fair. A client will get another shot at a lock within a short time window - this is less fair but greatly improves performance for highly contended locks. You can tune it or revert to older behavior. The best part was that Ari easily used the visualization tools to prove that locks weren’t fair.
- Does mutating state of clustered data require locking? - Yes. In the JVM there is no such protection - threads can modify a shared piece of data outside synchronization (creating a data race). In Terracotta, we will throw an exception (UnlockedSharedObjectException) if you try to modify clustered state outside a clustered lock.
- Are volatiles supported? - Yes. I believe the one caveat with Terracotta is that we require you to mutate volatiles in a synchronized block to create a Terracotta transaction under the hood, which you would not have to do without Terracotta. We could actually automatically insert those memory barriers but that prevents you from specifying lock scope at a more granular level to improve performance. One option we’ve discussed is allowing you to decide whether or not (via some config) to automatically insert volatile barriers. Then you could get it automatically but back off if you needed more granular locking.
- Is there an application to create the Terracotta XML config file? - Yes, the Terracotta Eclipse plugin does this. You can typically create the configuration without ever touching the xml. Other IDEs may be supported in the future (register, file and vote on JIRAs if you want it).
- If you have 150 app servers, how does that display in the admin console? - Each app server will show up as a client node in the tree. The Terracotta VisualVM plugin is a bit different as they make different display choices.
- Does the jvm go back to the server to obtain every lock? - No. It’s easy to show in the visualization tool that uncontended clustered locks are checked out to the node and do not need to talk to the server to otain the lock.
- Traditional system tracks how systems are given work and finish work. With Terraoctta how do we detect how jvms have gone away and/or whether they’ve finished work? - Terracotta has a JMX API that will fire events when either the Terracotta server or the cluster nodes leave or join the cluster. So you can use these as trigger events into whatever monitoring or API you need. For work completion, the data is actually coherent, so you can be guaranteed that as synchronized blocks complete, those memory transactions are persistent in the server state.
If you’re interested in seeing Terracotta at JavaOne, stop by booth 202 for a demo and to get all your questions answered. Or check out the Terracotta VisualVM plugin at the Sun booth.
