“top threads” plugin for JConsole

When working with large (server side) java application, sometimes it would be nice if you could look inside, to see what thread is taking up so much cpu time, and why. Something similar to the Unix top command, but then showing all threads in one (java) application, instead of all processes in the system.

When I was looking for such a monitoring application, I came accross the 2.0 version of MC4J that provides a “Thread Info” panel that displays threads together with CPU usage; exactly what I needed. Unfortunately, there is only an alpha release of this MC4J version, that is not yet perfectly stable. Moreover, the thread info panel doesn’t handle applications with large amounts of threads very well. As the source code of this version of MC4J is not (yet) publically available, this option turned out to be a dead end.

To my surprise, other applications with such functionality are hard to find. There are probably enough profiling applications that can do the job, but I wanted something simple, something JMX-based, that can used also to monitor applications running in production.

There is however something called JTop, which is a plugin for JConsole. It’s actually a demo for the new (since Java 6) JConsole plugin API, that does show CPU usage per thread. It’s fairly basic and only shows total CPU usage, which is not very usefull. You would expect that (after a year), somebody would have extended the demo to something more useful, but as I couldn’t find anything like that, I thought I should give it a try myself.

The result is a JConsole plugin that displays the top threads, sorted by cpu usage in the last sampling period. It also displays cpu usage history, and an average over the last 10 sampling periods.

 

To avoid ending up with an unresponsive user interface when monitoring applications with large number of threads, I took a few precautions. First of all, the plugin has it’s own refresh rate. It’s independent from the JConsole poll interval, which is 4 seconds by default. For applications with large amounts of threads, this is way too short: only retrieving all thread information can already take 4 or 5 seconds! Although you can change the JConsole poll interval with a command line option, I thought it would be more convenient to be able to change it from the monitoring panel. It’s default set to 10 seconds, which I think is reasonable in most cases. If you notice that cpu usage measurement takes too much of the application under test, just increase the sample period and see the RMI Connection thread that processes these request, sink in the list.

Another precaution was not to list all threads in the table. Displaying thousands of rows in a table is quite useless in any case, and I was afraid it would seriously harm performance. Eventually, diplaying that many rows turned out to be not much of a problem; I guess I still suffer from an prejudice with
respect to Swing performance…

Using MX4J also showed me that in a continuously refreshing table, it’s hard to select a thread in order to see it’s stacktrace. Therefore, in this plugin, tracing a thread is “sticky”: when you click a row in the table, the stacktrace of that thread is shown immediately and is refreshed with each new sample, until you deselect it or select another thread.

Even though having threads sorted by cpu usage is the logical thing to do, it’s not always convenient when you’re studying the results, as rows keeping moving with each refresh. To lock the rows to there current position, click the “fix order” button. The topmost rows (actually all rows with a non-zero cpu usage), will stay where they are. Rows that had a cpu usage of zero, but have a non-zero value in the next sampling periods, will appear just below these rows, to avoid that you oversee any thread that suddenly takes a large amount of cpu time.

You can run the plugin by downloading the jar-file and passing it to JConsole with the plugin option:
jconsole -pluginpath topthreads.jar. When JConsole connects, it should have a seventh tab named “top threads”.