Fighting the OutOfMemoryError

In: Java

1 Aug 2007

java logoSearching for memory leaks in an application is always a very difficult task. I experienced memory problems in a Java web application two months ago. After running some days an OutOfMemoryException was thrown because the application went out of heap memory.

Monitoring the application with the jconsole – which is an excellent tool for this task and it is included in the Sun JDK – showed that there was an amount of heap memory that could not be garbage collected and it slightly but steady increased over time. This meant that I had to search for the references that prevent the objects from getting cleared.

The first thing I did is to create multiple histograms of the heap while working with the application, destroying the session and starting garbage collection from jconsole. I did this by using jmap which is a small tool included in Sun JDK since version 5:

jmap -histo PID > histo.txt

Of course I had to fill in the PID of the VM running my application. Comparing the generated histograms I found that the number of instances of some objects in my application got more and more. These objects should have been garbage collected because they were just entity beans that should not be referenced anymore.

To go to the next level I used new features of Sun JDK 6. The jmap version of JDK 6 has more options than the version of JDK 5 and is much faster. Creating the histogram with version 5 took 58 seconds compared to about 1 (!) second with version 6. Additionally I used the :live option of jmap 6 to get only live objects in my histogram:

jmap -histo:live PID > histo.txt

Now that I identified some suspicious objects I had to find out what references them. There is another tool that comes with Sun JDK 6 for exactly that purpose: jhat. jhat is able to analyze binary memory dump files of a Java VM. Therefor I created a dump file of my application using the following command line:

jmap -dump:live,format=b,file=heap.bin PID

This creates a binary dump file of all live objects in the VM with the given PID and writes them in the specified file. To look into the dump I run jhat like

jhat -J-mx768m heap.bin

I had to add the memory option because otherwise jhat had not enough memory to display my 50MB dump file. After the successful start of jhat I could see the following on my shell:

Reading from heap.bin...
Dump file created Thu Jun 21 11:17:40 GMT+01:00 2007
Snapshot read, resolving...
Resolving 570010 objects...
Chasing references, expect 114 dots..................................................................................................................

Eliminating duplicate references..................................................................................................................

Snapshot resolved.
Started HTTP server on port 7000
Server is ready.

Pointing my browser to localhost:7000 brought up the following page:

jhat browser

This is a list of all the classes loaded in the VM. Now I searched for the classes I identified before in the histogram and on the details page of the class I clicked on the link named “All instances including subclasses”:

jhat instances

After selecting one of the instances I was able to see all other objects referencing this object instance:

jhat references

The classes in the screenshots are not the classes of my project – these are just examples – but in my project I found out that the problem were references from ThreadLocals. In my web application there some persistent objects and whole transaction/sessions were bound temporary to a ThreadLocal but the references were never set to null.

In every thread of the Tomcat server (which are 25 per default) references were hold to objects that could not be garbage collected. Under load the server created even more threads which were not closed early enough to avoid the OutOfMemoryError. To fix my problem I checked my code to make sure that after each request all the ThreadLocal references were cleared – and that resolved it.

8 Responses to Fighting the OutOfMemoryError

Avatar

prashant

August 3rd, 2007 at 4:16 pm

Hi,

very Good article but can you please suggest me what tool can be used other than jMeter to do performance testing please?

Thanks
Prashant

Avatar

prashant

August 3rd, 2007 at 4:20 pm

I mean for web based applications.

Avatar

chuck simpson

August 4th, 2007 at 8:54 pm

You might find SAP’s memory analyzer more useful than jhat. I did. It is an eclipse RCP application. It does what jhat does but seems quicker and uses less memory. Also once you have loaded a dump file the second time you open it it comes up immediately. You can open multiple dump files and get a delta between dumps with a single click. It is found at: https://www.sdn.sap.com/irj/sdn/wiki?path=/display/Java/Java+Memory+Analysis

Avatar

tompson

August 4th, 2007 at 9:21 pm

What’s wrong with jmeter prashat? I think it is very good tool for many testing purposes of web applications.

Thanks for the tip with the sap tool chuck, I will give it a try when I get back to Austria.

Avatar

John Borwick’s blog - Neat stuff John likes. » links for 2007-08-05

August 5th, 2007 at 10:22 am

[...] tOMPSON’s blog » Blog Archive » Fighting the OutOfMemoryError (tags: java outofmemoryerror debug debugging programming program example trace tracing memory out profiling development performance) [...]

Avatar

Ignacio Coloma

August 6th, 2007 at 9:22 am

Prashat, you may give a try to The Grinder. It is similar to jMeter, and supports phyton scripts.

Avatar

Martin Ankerl

September 10th, 2007 at 8:37 am

Nice article, this will be helpful! When you just need to find out object references during debugging, there is also a nice and easy way with Eclipse 3.3. See “Follow references” in the news & noteworthy:
http://download.eclipse.org/eclipse/downloads/drops/R-3.3-200706251500/whatsnew/eclipse-news-all.html

Avatar

hiutopor

September 18th, 2007 at 4:50 am

Hello

Very interesting information! Thanks!

G’night

Comment Form

Follow Me!

Photostream

    P1010008 (2)P1010026P1010028P1010021P1010009 (2)P1010033 (2)

RSStOMPSON's tweets