There are many discussion threads about PermGen memory problems around. Many of them concerning Eclipse running with Sun JDK 5 and greater - see my posts about JDK 5 and JDK 6. The solution for the problem with Eclipse was to set the amount of memory reserved for PermGen to a value high enough .
The same problem occurred to me when running large web applications with Tomcat inside a Sun VM. If there are to many classes in the project and the libraries it uses the PermGen memory setting had to be made.
It hit me hard last week when one of my Tomcat instances crashed with a PermGen space OutOfMemory error. I found out that it was caused by reloading one of the deployed applications inside the running Tomcat instance. It seemed like the classes were loaded again and the old ones were not garbage collected.
After doing some research I found many sites talking about similar problems. Some tried to adapt the size of the PermGen space but found out that this only delays the Problem. Others experimented with different application containers like Resin but the best solution seems to switch the VM.
To find out if this is true I set up the following test environment:
- Ubuntu 7.04
- Tomcat 5.5.23
- Sun JDK 1.6.0_01
I deployed Lambda Probe as the only web application and wrote a JMeter test that permanently reloads the application and calls the overview page of Lambda Probe. The following image shows the result I monitored with jconsole:
As you can see the memory usage of PermGen increased with every reload of the application until it reaches the point where the VM crashes with the OutOfMemory error. The default maximum PermGen amount of the Sun VM is 64 MB.
After switching the VM to the IBM JDK 5 I started the same JMeter test again. The jconsole showed a different result:
There seems to be no limit for PermGen space in the IBM VM. At about 300 MB my virtual machine started to swap a lot and everything got very slow so I stopped the test but there was no OutOfMemory error.
This is the second problem I had with web applications running on a Sun VM within two months. I am thinking about switching with all my application containers to the IBM VM. This problem makes the reload/redeploy feature of the Tomcat unusable.
September 13th, 2007 at 7:11 pm
servus tom! schon lang nichts mehr von dir gehört
meld dich wieder mal. weiter gehts in englisch:
i do not think that a switch to ibm jdk will solve the problem. a clean redeployment of your application should cause memory decline and not increasing. neither sun or ibm jdks did in your web application as you showed us.
imho your application trapped into a classloader memory leak. Frank Kieviet at sun wrote some excellent blog entries about this issue (indeed he also mentions how to fix such errors). either way its worth reading them:
This one is about the general issue of classloader leaks.
This one is about profiling and detecting them.
lg
September 14th, 2007 at 8:47 am
Hi stony, wow, these are two great blog entries, thanks!
In fact it was not “my” application I used for the test - it was lambdaprobe but after reading the two articles I could imagine that they made one of the mistakes described too. I will have to take another look into this issue.
September 23rd, 2007 at 4:59 pm
TomCat is known for having ClassLoading issues when doing reloads (I think there is even a section on this in the docs, saying “not recommended for production”). You will almost always see problems in conjunction w/ Log4J (since it is used by TomCat as well, see eg. here: http://issues.apache.org/bugzilla/show_bug.cgi?id=42172).
If I was you, I’d stay the hell away from IBM JVM - it plain sucks!
(and I disagree that increasing the PermGen endlessly is a good thing to do)
September 23rd, 2007 at 7:57 pm
I am using both JVM (Sun and IBM) in different projects in production environment and I have to say that the IBM JVM is not that bad. They both have their problems but there are situations when the IBM JVM is the better choice.
September 30th, 2007 at 12:42 am
Good post!
According to the articles that stoney found, it looks like there’s still hope for us all
I couldn’t help whipping up a summery: http://my.opera.com/karmazilla/blog/2007/09/29/return-of-the-permgen
Take care.
September 30th, 2007 at 8:31 am
the only thing that prevented me from experimenting with classloader leaks was the missing support for tracing classes in jhat. Fortunately, from JDK7b2 this issue will be fixed, as can be seen in SUN’s bug tracking system.
Another interesting thing about jhat I tumbled over, is the possibility to create custom reports by using jhat’s OQL interface. A. Sundararajan’s blog has some really good posts on that topic:
*) querying java heap with OQL
*) jhat’s javascript interface
October 1st, 2007 at 10:01 pm
This problem is not only for the tomcat.
I am running an application on the Apache Geronimo (jetty based) with JDK 5, after one or two days, jvm gives same error.
October 2nd, 2007 at 12:05 am
This is a memory leak issue. It affects all app servers. It is caused by references from objects outside the webapp classloader back into the classes loaded by the webapp. One such reference prevents that class from being GC’d, which prevents the classloader from being GC’d, which prevents all classes under it from being GC’d.
These references fall into two types: a) classes that register with system classes (e.g. a JDBC driver in your webapp that registers with the system DriverManager) and b) ThreadLocals held by the connection pool. The ones in class a) are usually fixable by running cleanup code when your webapp context is destroyed. The ones in b) are much harder and this basically represents a bug/feature in both the ThreadLocal class and the fact that app servers don’t have a convenient way to dedicate Threads to webapps and recreate them as part of reloading the web app. Careful use of ThreadLocals can alleviate this, but finding offenders is pretty hard, even with a good profiler.
October 2nd, 2007 at 12:08 am
Here is something I discovered a while ago which you might find useful
http://blogs.boxysystems.com/2007/4/11/tomcat-v5-0-28-memory-leak-exposed
October 2nd, 2007 at 12:17 am
hi there,
tomcat 6.x fixed this bug (among others). try tomcat 6 and let us know if that worked.
BR,
~A
October 2nd, 2007 at 8:08 am
[...] java.lang.OutOfMemoryError: PermGen space [...]
October 3rd, 2007 at 9:57 am
I think I will run the test again with Tomcat 6.x as soon as I get a time frame
October 11th, 2007 at 9:42 am
[...] response to my post about OutOfMemory PermGen errors in Tomcat with Sun JDK brought up some very interessting things. I learned a lot from Frank Kieviet’s blog about [...]
April 23rd, 2008 at 1:46 am
[...] passing command line arguments to the JVM or changing the size of the PermGen space, others end up recommending using a VM from BEA or IBM, all without [...]