If you just want a quick fix to the problem without understanding the theory, jump to part IV introducing the ClassLoader Leak Prevention library.
A couple of years ago, there was an article on the Confluence site of the Spring framework written by Magnus Alvestad, titled “Memory leaks where the classloader cannot be garbage
collected” (originally located here/here). Apart from a short description of the problem with classloader leaks, it also contained a list of known offenders; practices and third party libraries known to cause classloader leaks, either by incorrect use or as a bug.
Even though not by far exhaustive, an article like that was a good reference – a place to start looking if you experienced
java.lang.OutOfMemoryError: PermGen space. Unfortunately it seems that page is not available any more. My hope for this post is that it may serve that purpose instead. For that to come true, I also need your help. If you know of a public Java library that does, in a current or previous version, cause classloader leaks, please comment on this post. I intend to keep the list updated. (Please note that this is not an invitation for support requests, asking me whether a particular library does leak. I want you to report leaks that are confirmed, either by yourself or that there is an online bug report / mailing list discussion. References are preferrable!)
Let’s start with the leaks that aren’t necessarily bugs, but failure to follow guidelines or best practice. It may also be API:s that have the potential to cause classloader leaks, while they also provide a means to avoid them.
As we talked about in the previous post, you should always call
remove() on a
ThreadLocal in a
In part III of this series we talked about the importance of making sure threads started within your web app stop executing when the application is redeployed.
JDBC driver included in WAR
Your JDBC driver will be registered in
java.sql.DriverManager, which means that if you include your JDBC driver inside your web application, there will be a reference to your webapps classloader from system classes (see part II). The simple solution is to put JDBC driver on server level instead, but you can also deregister the driver at application shutdown.
Logging frameworks such as Apache Commons Logging (ACL) – formerly Jakarta Commons Logging (JCL) – log4j and
java.util.logging (JUL) will cause classloader leaks under some circumstances.
Apache Commons Logging will cause trouble if the logging framework is supplied outside of the web application, such as within the Application Server. In such a case, you need to add a bit of cleanup code to the
ServletContextListener we’ve talked about:
There is an article about this on the Apache Commons Wiki. It is also mentioned in the guide and FAQ.
When it comes to log4j, you can achieve the same kind of leak with some configurations. I’m not sure if calling
org.apache.log4j.LogManager.shutdown() in the cleanup
ServletContextListener helps, but it’s probably a good idea anyway, at least if there is only a single web application running on the server.
With log4j2, it seems the shutdown mechanism is a bit more complicated. I hope to get back with a more proper solution, but for now see this Stac kOverflow answer.
java.util.logging (JUL), it will always be outside the web application, since it is part of the JDK. With it you can cause classloader leaks by creating custom log Levels inside your web app. This is what Frank Kieviet uses as example in his blog post on classloader leaks. It seems also that JBoss Logging does exactly that if backed by JUL, and thereby triggers such a leak if loaded within your web app. See report here.
There is also a thorough article about the problem with logging frameworks and classloaders here. As a general recommendation, SLF4J supposedly helps preventing at least some of these problems.
The Java Bean introspection has a cache with strong references, that need to be cleared by calling
in the cleanup
ServletContextListener. If you don’t want to create your own, you can use
org.springframework.web.util.IntrospectorCleanupListener from the Spring framework.
Custom property editor
If a property editor loaded within the web application (or a property editor for a class loaded in the web application) is registered by calling
java.beans.PropertyEditorManager.registerEditor() it needs to be deregistered at application shutdown, or it will cause classloader leaks. Deregistering can be achieved by calling
java.beans.PropertyEditorManager.registerEditor() again with the same
null as the second argument (
java.net.Authenticator loaded in your web application and registered with
java.net.Authenticator.setDefault() must be unregistered at application shutdown, or it will cause leaks.
java.net.ProxySelector loaded in your web application and registered with
java.net.ProxySelector.setDefault() must be unregistered at application shutdown, or it will cause leaks.
java.security.Provider loaded in your web application and registered with
java.security.Security.addProvider() must be unregistered with
java.security.Security.removeProvider() at application shutdown, or it will cause leaks. However, there is also a problem with
javax.crypto.JceSecurity that holds a couple of static caches. So in case you’ve used a custom
java.security.Provider for cryptographic operations, this will cause a leak even if you call
Custom MBeans registered in the
ManagementFactory.getPlatformMBeanServer().registerMBean()) needs to be unregistered at application shutdown, or they will cause leaks.
MBeans/MXBeans implementing the
javax.management.NotificationBroadcaster interface – or the
javax.management.NotificationEmitter sub-interface – may have
javax.management.NotificationListeners added to them, in combination with a
javax.management.NotificationFilter and/or a handback object. If any of these three are loaded by your web application, you would need to call
removeNotificationListener() on shutdown to prevent leaks. Apache Pig has such a leak reported, but users are adviced to remove Pigs
SpillableMemoryManager from the MemoryMXBean on application shutdown.
ThreadGroups are in a hierarchy with the “system”
ThreadGroup at the top. Children are added to a parent upon creation, and parents will keep a reference to it’s child
ThreadGroups, until the child is destroyed by a call to
destroy(). This means that if
java.lang.ThreadGroup is subclassed, and that subclass is loaded inside your application and instantiated, then there will be a strong reference to your applications classloader until that
Among the API:s that are known to cause classloader leaks, without providing a proper cleanup/workaround, are a couple of JDK classes with the habit of keeping a reference to the
contextClassLoader of the
Thread that first calls them. The solution in this case is to make sure these methods are called once, with some other classloader – such as
ClassLoader.getSystemClassLoader() – as
contextClassLoader. Preferrably we put this code in the
contextInitialized() method of our
ServletContextListener. See Tomcats JreMemoryLeakPreventionListener class for more details.
Apache ActiveMQ registers
org.apache.activemq.util.StringArrayEditor as a property editor for
java.beans.PropertyEditorManager (by a
static block in
org.apache.activemq.util.IntrospectionSupport), but provide no means of deregistering it. Reported here.
Apache Axis leaks classloaders because of uncleared
ThreadLocal, at least version 1.4, as we saw in part IV.
Some versions of Batik SVG Toolkit (at least 1.5 beta 4 up to 1.7), leaves unterminated threads, as we saw in part III. It has been reported but at the time of this writing there is still no fix.
Apache Commons Pool / DBCP
Apache Commons Pool, which is used by Apache Commons DBCP, has a feature to automatically evict idle objects, which use a background thread running in the applications classloader. Earlier versions suffered from a bug, that did not allow the thread to be stopped properly (missing synchronization or
volatile, like talked about in part III). See this blog post. It seem that in recent versions you can simply call
org.apache.commons.pool2.impl.GenericObjectPool.close() at application shutdown. You can also turn off the idle evict feature at any time by calling
org.apache.commons.pool2.impl.GenericObjectPool. with a negative value.
Apache Commons HttpClient
Apache Commons HttpClient has a
MultiThreadedHttpConnectionManager on which you may need to call the static
shutdownAll() method in order to stop its
ReferenceQueueThread, in case it has been loaded in your web app. This leak may be triggered by the Jersey
Apache CXF may set
org.apache.cxf.transport.http.CXFAuthenticator as the default
java.net.Authenticator; see above. (Thanks to Arild Froeland for the report!) Reported here.
Bean Validation API / JSR 303
The Bean Validation API / JSR 303 will leak classloaders, if the API is at the application server level while the implementation, such as Hibernate Validator, is included inside your web application. More details in part II.
CGLIB / Hibernate / Spring / JBoss / Apache Geronimo
CGLIB, used by Hibernate, Spring etc, has had a bug in it’s proxy code, with uncleared
ThreadLocals (see part IV). Bug report on Hibernate. Bug reports on Spring. Bug report on JBoss. Bug report on Apache Geronimo.
dom4j has uncleared
ThreadLocals (see part IV). Reported here. That bug report is still “open”, but on other parts of the net report it to be fixed in 1.6.1.
Two very similar issues can be triggered by creating a DOM document, and then either normalize it or serialize it, such as the following code
import javax.xml.parsers.DocumentBuilderFactory; import org.w3c.dom.Document; import org.w3c.dom.ls.DOMImplementationLS; ... Document document = DocumentBuilderFactory.newInstance() .newDocumentBuilder().newDocument(); document.normalizeDocument(); // or DOMImplementationLS implementation = (DOMImplementationLS)document.getImplementation(); implementation.createLSSerializer().writeToString(document);
The reason for the potential leak is that both
com.sun.org.apache.xml.internal.serialize.DOMSerializerImpl have a
static field named
abort, each containing a
RuntimeException whose stack trace may contain a reference to the first class that invoked the normalization/serialization. Reported here.
contextClassLoader of the thread loading the
com.sun.jndi.ldap.LdapPoolManager class may be kept from being garbage collected, since it will start a new thread if the system property
com.sun.jndi.ldap.connect.pool.timeout is set to a value greater than 0.
The EclipseLink JPA implementation has had classloader leaks. According to this bug report fixed since version 1.1.
GeoTools is reported here to have unending thread. As of version 2.6.2 you should call
org.geotools.util.WeakCollectionCleaner.exit() in your cleanup.
Googles IoC/DI framework Guice seems to have had several classloader leaks, at least some of which are not resolved at the time of this writing. Reports here, here and here.
Groovy can cause classloader leaks according to this bug report, which is still “open”.
Hessian binary web service protocol has suffered from uncleared
ThreadLocals (see part IV). Bug reported here. Should be fixed since version 4.0.23 shipped with Resin.
iCal4J seems to have sufferend from uncleared
ThreadLocals (see part IV). More info here.
Infinispan has had a number of reports of uncleared
ThreadLocals. Two examples here and here, but there are more if you look at “Similar Issues” or search Jira yourself.
IntrospectionUtils copied from Tomcat (
org.apache.tomcat.util.IntrospectionUtils) to Apache Commons Modeler (
org.apache.commons.modeler.util.IntrospectionUtils) keeps a strong reference cache. After this was reported as a bug, a static
clear() method has been added. Make sure to call it on application shutdown.
In case Jackson is included within the application server but used in a web application,
com.fasterxml.jackson.databind.type.TypeFactory.defaultInstance().clearCache() should be called when that application is shutting down to avoid leaks. Reported here. Report against Wildfly, where Jackson is included, here.
Java Advanced Imaging (JAI)
Java Advanced Imaging (JAI) library can cause classloader leaks with registered shutdown hooks, as we saw in part II. Bug report here – still “open”.
The first calll to
java.awt.Toolkit.getDefaultToolkit() will spawn a new thread with the same
contextClassLoader as the caller.
I’ve seen reports on the net about PermGen errors when using older versions of Javassist for example with Hibernate. There are lots of issues in Javassist JIRA; start here and see “Similar Issues”. Since the issues have different fixed versions, you better use the latest version.
Java Cryptography Architecture (JCA) /
According to Tomcat documentation, a Token poller thread with the same
contextClassLoader as the caller, will be created “under certain conditions” when Java Cryptography Architecture is initialized, for example when a
MessageDigest is created.
Java Server Faces 2
The JSF API, more precisely
javax.faces.component.UIComponentBase, contains a cache that may cause classloader leaks in case the API is at the application server level. More information in this bug report.
sun.awt.AppContext.getAppContext() / GWT
There will be a strong reference to the classloader of the calls to
sun.awt.AppContext.getAppContext(). Note that Google Web Toolkit (GWT) will trigger this leak via its use of
javax.imageio. Another way to trigger this leak is by creating a
new javax.swing.JEditorPane("type", "text").
sun.misc.GC.requestLatency(long), which is known to be called from
javax.management.remote.rmi.RMIConnectorServer.start(), will cause the current
contextClassLoader to be unavailable for garbage collection.
javax.net.ssl package is used together with a keystore that contains certificates that have unparseable extensions, there will be a reference from system classes to the exception occurring while trying to parse the extension. The
backtrace (stacktrace) of that exception is likely to contain references to the code that called the
javax.net.ssl classes. In a servlet environment this means the first webapp instance to trigger
javax.net.ssl will be prevented from being garbage collected. Reported to Oracle here. Prevented since version 2.1.0. Thanks to “CptS” for the report!
javax.security.auth.Policy.getPolicy() will keep a strong static reference to the
contextClassLoader of the first calling thread.
javax.security.auth.login.Configuration will keep a strong static reference to the
Thread from which the class is loaded.
JGroups may cause classloader leaks due to undestroyed custom
ThreadGroups. At the time of this writing this issue has been fixed, while this one has not.
The no longer maintained LambdaJ seems to suffer from uncleared
ThreadLocals in some cases.
Logback causes classloader leaks when using
SocketAppender according to this report, in which is it not really clear whether it has been fixed.
Logback also seems to have suffered from uncleared
ThreadLocals (see part IV), as of this report. However that should be fixed since version 0.9.26.
javax.xml.bind.DatatypeConverterImpl in the JAXB Reference Implementation shipped with JDK 1.6+ will keep a static reference (
datatypeFactory) to a concrete subclass of
javax.xml.datatype.DatatypeFactory, that is resolved when the class is loaded (which I believe happens if you have custom bindings that reference the static methods in
javax.xml.bind.DatatypeConverter). It seems that if for example you have a version of Xerces inside your application, the factory method may resolve
org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl as the implementation to use (rather than
com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl shipped with the JDK), which means there will a reference from
javax.xml.bind.DatatypeConverterImpl to your classloader.
If the Mojarra JSF implementation is provided by the application server, and you have JSF components included in your .war file, Mojarra may cause classloader leaks by keeping references to the components when application is redeployed. See bug report here.
If JAXB implementation EclipseLink MOXy is included on the application server level, as is the case for at least some WebLogic versions, and then used within a web application, it may cause leaks. See bug report here.
ThreadLocals. You can read about the details in this blog post. The bug report states that it’s fixed (supposedly in 2005), but not from which version.
MVEL expression language enginge has problems with uncleared
ThreadLocals. The problem was adressed but version 2.0.19 still leaks, due to a static block adding an MVEL class as a
ThreadLocal value. Current bug report here.
OpenOffice Java Uno RunTime (JURT)
The JURT (Java Uno RunTime) library in the OpenOffice UDK contains a peculiar leak. In an effort to avoid
OutOfMemoryErrors(!) caused by long-running finalize methods, they created the
com.sun.star.lib.util.AsynchronousFinalizer class, which holds a queue of jobs to be executed by a
Thread. The problem is that this thread is never allowed to finish, as we talked about in part III. What makes this particularly tricky is that we don’t have any handle to the thread, and not least the fact that jobs are put on queue, and the thread is started if not yet running, from the
finalize() method of other objects. That is, the thread may be started only when other objects are garbage collected, which may – at least in theory – not be until after your web application has been already unloaded! This problem has been reported here.
Oracle JDBC driver, at least in some versions (such as ojdbc6), have a timeout detection thread,
oracle.jdbc.driver.OracleTimeoutPollingThread, that according to reports has as its context classloader the classloader of the web application from which the first JDBC connection is requested, even if the driver itself resides on the server level. It seem this can be prevented by loading the class
oracle.jdbc.driver.OracleTimeoutThreadPerVM using the system classloader before any JDBC connections are opened, which can be achieved in
contextInitialized() of our
ServletContextListener. (Thanks to Hal Deadman for the report!)
Postgresql JDBC driver will under some circumstances start timer threads with names prefixed
PostgreSQL-JDBC-SharedTimer-. These threads may prevent your classloader from being garbage collected, presumably due to the inheritance of the context classloader and access control context. This has been reported here, but at the time of writing it is unclear if this is actually fixed or not. (Thanks to Hal Deadman for the report!)
SAAJ (SOAP with Attachments API for Java)
In the Sun/Oracle implementation of SAAJ, the class
com.sun.xml.internal.messaging.saaj.soap.EnvelopeFactory has a cache of
javax.xml.parsers.SAXParsers per context classloader. The classloader key is a proper weak reference, but in case the
SAXParser or any of its properties/attributes are loaded by the very same classloader, it means there is a strong reference path from a system class to this classloader, which will prevent it from being GC:ed. For an example, where this could be avoided by resetting the
SAXParser after use, see here. (Thanks to Samuel Brezáni for the report!)
Up to Java 1.4.2, serialization of classes loaded by your web apps classloader would cause reference to your classloader most likely to be kept by internal JDK caches. More info in this blog post. Bug report here.
Springs use of
InheritableThreadLocal has caused leaks. Fixed since 1.2.9 / 2.0.1.
Loading the class
sun.java2d.Disposer will spawn a new thread with the same
contextClassLoader. More info. Fixed in Java 7.
In order to close reusable HTTP 1.1 connections after a timeout,
sun.net.www.http.KeepAliveCache starts a thread which will keep a strong reference to the classloader from which the HTTP connection was initiated (through a
ProtectionDomain). See more info in this blog post. According to Magnus Alvestads article, the thread will terminate eventually, which may cause intermittent classloader leaks. If you have control over the server end you could add “
Connection: close” to the HTTP responses. On the client end you can disable keep alive using
Unified Expression Language
Up until version 2.2, the
javax.el.BeanELResolver class of the Java Unified Expression Language /
javax.el API kept a static strong reference cache with bean classes that had been introspected. This was reported as a leak and has since been fixed in version 2.2.4+ by using soft references in the cache. For currently released versions, the authors have been kind enough to provide a
purgeBeanClasses() method to clear the cache, however this method is both private and non-static… (The JavaDoc of the method is a bit of a funny read, as it suggests the authors realized the problem, but failed to provide a proper solution.)
The caching mechanism of
JarURLConnection can prevent JAR files to be reloaded. See this bug report. It is not entirely clear whether this will actually leak classloaders.
The classloader of the first thread to call
DocumentBuilderFactory.newInstance().newDocumentBuilder() seems to be unable to garbage collection. Is it believed this is caused by some JVM internal bug.
Your Application Server
Yes, unfortunately the Application Server itself can be the cause of your classloader leaks. Tomcat (reports here, here, here), Jetty (here) and Resin (reports here, here, here, and more) has been know to suffer from such bugs, but there may be others as well. (There is a report on GlassFish here, but it’s not clear whether the error was actually within GlassFish.)
Pick the right Garbage Collector
It seems that for some reason, the default Garbage Collector of the Sun/Oracle JVM is not your best bet when it comes to freeing up permanent generation space. Assuming you have a multi core/multi CPU server (and who doesn’t these days?) I suggest you consider the Concurrent Mark and Sweep (CMS) Garbage Collector instead (but make sure you read up on the implications first). The CMS Garbage Collector however needs to be explicitly told to unload classes from PermGen space. Use these JVM options:
Depending on your JVM version, you may also need
-XX:+CMSPermGenSweepingEnabled. Try them out and keep an eye on the console/log file during startup to see if the JVM accepts them.
For further reading on different Garbage Collectors, see here.
Links to all parts in the series
Part I – How to find classloader leaks with Eclipse Memory Analyser (MAT)
Part II – Find and work around unwanted references
Part IV – ThreadLocal dangers and why ThreadGlobal may have been a more appropriate name
Part V – Common mistakes and Known offenders
Part VI – “This means war!” (leak prevention library)