Classloader leaks VI – “This means war!” (Leak Prevention library)

This post is intended to conclude the series about classloader leaks, and I want to use it to declare war! (Not as in Web application archive though…)

I have decided to pursue the fight against classloader leaks, and I invite you to join me. For this purpose I have created a project on GitHub called classloader-leak-prevention. The project consists of two parts.

Classloader leak protection listener

First and foremost there is a component for you to add to your web application, that intends to remove and work around as many of the known issues as possible. This will allow us not to depend on bugs being fixed in third party libraries. Yes, this is somewhat like what Tomcat has built in, but my component covers cases that Tomcat currently does not. Another major advantage is that my component is Application Server independent.

It should be as easy as adding a JAR or .java file, configuring a ServletContextListener in web.xml, and you should be protected against java.lang.OutOfMemoryError: PermGen space caused by your app leaking classloaders (you could still run out of PermGen however, or other apps on the same server may leak). In due time, I hope that parts of it will be configurable to your needs (check back here for updates). If it is not configurable enough, feel free to subclass or create your own GitHub fork.

To configure the main prevention mechanism, just add the component to your project and insert this into your web.xml:

<listener>
  <listener-class>
    se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor
  </listener-class>
</listener>

It makes sense to keep this listener “outermost” (initializing first, destroying last), so you should normally declare it before any other listeners in web.xml.

Maven

The library is available in Maven Central with the following details:

<dependency>
  <groupId>se.jiderhamn</groupId>
  <artifactId>classloader-leak-prevention</artifactId>
  <version>1.15.2</version>
</dependency>

Download

Non-Maven users can download the JAR with the current version (1.15.2) of the project » here «.

Configuration

The context listener has a number of settings, see the readme on GitHub.

Classloader leak detection / test framework

Another part of the project, is a framework that allows the creation of JUnit tests, that confirms classloader leaks in third party APIs. It is also possible to test leak prevention mechanisms to confirm that the leak really is avoided.

Read more about this on GitHub.

License

This project is licensed under the Apache 2 license, which allows you to include modified versions of the code in your distributed software, without having to release your source code.

Links to all parts in the series

Part I – How to find classloader leaks with Eclipse Memory Analyser (MAT)

Part II – Find and work around unwanted references

Part III – “Die Thread, die!”

Part IV – ThreadLocal dangers and why ThreadGlobal may have been a more appropriate name

Part V – Common mistakes and Known offenders

Part VI – “This means war!” (leak prevention library)

Presentation on Classloader leaks (video and slides)

  • Arild Froeland

    Kudos :-)nnI had to add these two to clean up after Apache CXF and the Oracle JDBC drivernn public void contextDestroyed(ServletContextEvent servletContextEvent) {n super.contextDestroyed(servletContextEvent);nn clearJavaNetAuthenticator();n removeOracleMBean();n }nn private void clearJavaNetAuthenticator() {n // Used by org.apache.cxf.transport.http.CXFAuthenticatorn java.net.Authenticator.setDefault(null);n }nn private void removeOracleMBean() {n try {n MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();nn int hashcode = Thread.currentThread().getContextClassLoader().hashCode();n String hexString = Integer.toHexString(hashcode);n ObjectName objectName = new ObjectName(“com.oracle.jdbc:type=diagnosability,name=*@” + hexString);n Set objectNames = mBeanServer.queryNames(objectName, null);n for (ObjectName o : objectNames) {n info(“Unregistering Oracle MBean, ” + o);n mBeanServer.unregisterMBean(o);n }n } catch (MalformedObjectNameException e) {n error(e);n } catch (MBeanRegistrationException e) {n error(e);n } catch (InstanceNotFoundException e) {n error(e);n }n }n

    • Thank you for the report Arild. I have created a test case that confirms the Apache CXF leak, and included a (slightly more sofisticated) general fix for such leaks.nnI have also started looking into the Oracle JDBC MBean leak, but I’m not sure when I’ll find the time to finish.

      • Arild Froeland

        I can confirm this works as expected, it’s being released, but it does not work for us.nThe reason is that we are hotswapping the application and the new instance of CXF sets the default handler.nAs a consequnce isLoadedInWebApplication is false. Which is correct.nIf we don’t hot hotswap, i.e stopping and starting the webapp, then it works.nnThe problem is basically that it’s the default authentication handler for the JVM, not the webapp.nLuckily for us we don’t need the default authenticator so we can override and always use Authenticator.setDefault(null) as a workaroundnRef http://cxf.547215.n5.nabble.com/Re-Discuss-change-http-authorization-handling-to-be-strategy-based-td3266405.htmln

        • It sounds to me, as if you are talking about a case where the application server supports application versioning, so that it may still be serving requests in the old instance, while the new instance is being deployed.nThat is, CXF in instance 2 of the application may call Authenticator.setDefault() before instance 1 is being shut down and the Authenticator cleared. If that is the case, I guess CxfAuthenticator from instance 2 will wrap CxfAuthenticator from instance 1, thus keeping a reference to the classloader of the old instance.nnIs that the case you are seeing?

          • Arild Froeland

            A bit late, but .. :-)nYes, I assume that is what happens.nnDue to port conflicts (embedded ActiveMQ) and jmx bean conflicts etc we are now stopping the application completely so we don’t have the problem anymore.n

          • Hi again Arild. I recently ran into this same issue myself, and think I have found out what is going on, at least enought to file a bug report: https://issues.apache.org/jira/browse/CXF-5442

            I also added workaround for this issue in my library, and hope to release a new version shortly.

    • Instead of focusing on Oracle JDBC, I created a test case for leaks caused by custom MBeans, and then I created a generic fix for such leaks. There is an updated JAR available here.nnFrom what I can see in the Oracle JDBC driver, this should cover that problem too, although I have not tested it. Feel free to verify.nn(Otoh, JDBC driver should not be placed inside the application, and moving the JDBC driver to server level should have avoided this leak anyway?)

      • Arild Froeland

        This one also works :-)nnRegarding the JDBC driver.nWe run our applications using an embedded version of Jetty.nI.e one jetty == one webappnI have actually never seen a reason for using multiple applications in a servlet/web container.nIt’s so much easier to only use one application, and the overhead is not a problemnPlus different versions of the database sometimes need different versions of the driver so sharing is sometimes problematic.

  • Hal Deadman

    I think I have added this to about 30 different web applications and it seems like I am seeing fewer memory leaks and getting more hot deployments without memory issues. I am suprised this listener isn’t getting more publicity but maybe its just not that hot of a topic?. Maybe its too early to post a follow-up to your “This means war” post that is titled “Mission Accomplished” but I think you are pretty close. nnI have been trying to think of a way where this listener could register something at the system level that could keep track of all the classloaders associated with deployed apps (that use this listener). If it had some type of reference to the classloaders (provided by the listener during deployment) that wouldn’t prevent them from getting GC’d but would be able to tell if they had been GC’d, then you could query the system level class (e.g. via jconsole) to see which applications’ classloaders were still around. The system level class could keep track of them by name and time of deployment, so if an application that had been deployed several times had multiple instances around, you would know you had a problem with that application. If this could be done in a way that was application server independent and followed the single-class/no dependency pattern of your listener, then it would seem to accomplish most of what Arit was trying to do but with a lot less code. nnThis code would need to be deployed in the system classpath which I normally try to avoid but it might be warranted in this case. If it ran as a JVM shutdown hook after applications were undeployed (already done by app server on shutdown) then it could try to force a permgen collection and report any applications that were still around. nnDo you think something like this would work? Also, do you have any experience with the Java 7 garbage collector with respect to it doing permgen collections?n

    • Sam Stephens

      Yeah this project is fantastic. Has dropped my restarts from 5 times a day to once a week. I’d love to contribute to the project but I cannot figure out how to contact the author!

      • Hi Sam. Glad you appreciate the project and want to contribute. The prefered way is by creating Pull requests on GitHub. Not everyone uses Git (yet…), so I should probably consider a spam safe way of sending me patches.nnMeanwhile I will send an e-mail to your Disqus address, visible to me as a site admin, that you can reply to.

    • Martijn van Tilburg

      Do you mean G1 collector where you say Java 7 garbage collector? The G1 collector in Java 7 is not suitable if your application relies on serious class loading and unloading. Please note that starting with Java 8 the PermGen will disappear.

  • Vinicius

    Congratulations for developing such a useful listener! This save us a lot of time of restarting web containers like Tomcat. Thank you very much!

    • Thanks for the suggestion and link. Just want to let you know that I have started the process of publishing to Maven Central. Will of course get back with more info once it’s done.

    • The library should be available in Maven Central now.nnn se.jiderhamnn classloader-leak-preventionn 1.5.2nn

  • Gilberto

    Hi, Mattias! Thank you for this useful API. Just one advice, when putting the classloader-leak-prevention API on pom.xml file it brings together the el-api.2.2.1, thus causing conflict on tomcat server. We have had to exclude it!

    • Thanks for the heads up! I just released 1.7.1 where the scope of el-api is changed to test, as with all other dependencies.

  • Viktor Chuhra

    Thanks for your post!nUsing this API I was able to resolve the problem with memory leaking in my project. The reason was the plugin framework – jspf (https://code.google.com/p/jspf/). It creates ExecutorService for its own purpose and doesn’t close it properly. So using this code I was able to narrow down the problem and find the cause using profiler. Thanks again! 🙂

  • Pingback: Fighting the PermGen hell | Liferay()

  • seanf

    I couldn’t find a mailing list, so I guess this is the place to ask…

    It should be safe for ClassLoaderLeakPreventor to log using java.util.Logging (as a JDK class), shouldn’t it?

    • The library has been designed so that you may easily subclass and override the log methods to use the means of logging that you prefer and consider safe.

      I’m not sure JUL is safe enough as the default, considering that AFAIK you may bridge JUL over other APIs (such as SLF4J) potentially to some unsafe library.

      But to answer your question, yes with the default setup I believe JUL should be safe.

      • seanf

        Ah yes, attaching a bridging Handler to the root Logger might well cause a leak. Hmm. Perhaps the library could check for leaking Handlers attached to Loggers, but it could happen to any Logger, not just root, so where would you stop?

        I was able to redeploy my JBoss EAP 6 app dozens of times with the help of ClassLoaderLeakPreventor, but then I tried fetching the app home page in between redeploys. There must be some lazily-loaded memory leak still in there, so I’d better read this series properly…

        Thanks for implementing a great idea, and for the reply.

  • BC

    Hi
    Mattias,

    This sounds terrific. We run out of permGen space after displaying numerous BIRT
    reports. Your solution sounds just like what we need. I have added the listener
    to web.xml and added classloader-leak-prevention-1.9.3.jar to WEB-INF/lib. I
    get the following error when I run

    INFO [STDOUT] ClassLoaderLeakPreventor: stopThreads =true
    INFO [STDOUT] ClassLoaderLeakPreventor: stopTimerThreads = true
    INFO [STDOUT] ClassLoaderLeakPreventor: executeShutdownHooks = true
    INFO [STDOUT] ClassLoaderLeakPreventor: threadWaitMs= 5000 ms
    INFO [STDOUT] ClassLoaderLeakPreventor: shutdownHookWaitMs= 10000 ms
    INFO [STDOUT] ClassLoaderLeakPreventor: Initializing context by loading some known offenders with system classloader
    ERROR [[/RapidBIRTWebViewer]] Exception sending context initialized event to listener instance of class se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor
    java.lang.Error: javax.xml.datatype.DatatypeConfigurationException: Provider
    org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl
    not found
    at javax.xml.bind.DatatypeConverterImpl.(DatatypeConverterImpl.java:891)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:188)
    at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.contextInitialized(ClassLoaderLeakPreventor
    .java:262)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669)

    ——

    org.apache.xerces_2.9.0.v201101211617.jar is I the same directory and contains datatype.DatatypeFactoryImpl. Any ideas why it can’t find the class?

    Thanks
    BC

    • Thanks for the report. I notice that I had added a comment in the code that Class.forName(“javax.xml.bind.DatatypeConverterImpl”) may throw java.lang.Error, but for some reason the catch block only caught ClassNotFoundException. I will fix that to the next version.

      • BC

        Any idea when you will create the next version?

        • There are some other, larger changes that need some more testing first, but I’m hoping to get it out later this week.

        • Version 1.10.0 has just been released!

          • BC

            Hi Mattias,

            I have the new version of classloader-leak-prevention jar,
            org.apache.xerces_2.9.0.v201101211617.jar is in the same directory with the DatatypeFactoryImpl class. I get the same error

            07 May 2014 12:08:43,334 INFO [STDOUT]
            ClassLoaderLeakPreventor:
            shutdownHookWaitMs = 10000 ms

            07 May 2014 12:08:43,334 INFO [STDOUT]
            ClassLoaderLeakPreventor: Initializing context by loading some known offenders
            with system classloader

            07 May 2014 12:08:43,974 INFO [STDOUT]
            java.lang.Error: javax.xml.datatype.DatatypeConfigurationException: Provider
            org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl
            not found

            07 May 2014 12:08:43,974 INFO [STDOUT] at
            javax.xml.bind.DatatypeConverterImpl.(DatatypeConverterImpl.java:891)

            07 May 2014 12:08:43,974 INFO [STDOUT] at java.lang.Class.forName0(Native Method)

            07 May 2014 12:08:43,974 INFO [STDOUT] at
            java.lang.Class.forName(Class.java:188)

            07 May 2014 12:08:43,990 INFO [STDOUT] at
            se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.con
            textInitialized(ClassLoaderLeakPreventor.java:271)

            Any ideas?
            Thanks BC

          • I’d say that is expected behaviour, since it will be logged as a warning. Do you see any signs that it is actually not working? Are there any further ClassLoaderLeakPreventor logs after this?

  • Хүрэлхуяг

    Can I register this listener server level instead of include in every webapp?

    • I’m not aware of that being supported by the servlet spec, but I believe that in some app servers (such as Resin) you could achieve that by adding a server level path to the web apps classpaths via server level config.

      I would suggest that you file a feature request with your app server and ask them to integrate the ClassLoader Leak Prevention into the servers classloader management itself. 🙂

  • Pankaj Khurana

    I am facing following issue:

    02-Feb-2017 23:09:07.669 SEVERE [localhost-startStop-2] org.apache.catalina.core.StandardContext.listenerStart Error configuring application listener of class se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor
    java.lang.InstantiationException: se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor
    at java.lang.Class.newInstance(Class.java:427)
    at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:119)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4775)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:940)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1816)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.lang.NoSuchMethodException: se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.()
    at java.lang.Class.getConstructor0(Class.java:3082)
    at java.lang.Class.newInstance(Class.java:412)
    … 14 more

    • What version of the library are you using (1.x or 2.x)?
      Please notice that the name of the listener class has changed in 2.x, so you’d need to update your web.xml when upgrading. New name is se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventorListener

      • Pankaj Khurana

        I think I was wrong. Now I corrected it. I am having servlet 3.0

        env, Do I just need to include following dependency in my pom.xml?

        se.jiderhamn.classloader-leak-prevention
        classloader-leak-prevention-servlet3
        2.2.0

        Is there anything else I need to do like defining listener in web.xml?

  • Tim Clotworthy

    My understanding is that ClassLoaderLeakPreventor protects against java.lang.OutOfMemoryError: PermGen space caused by your app leaking classloaders. Our memory fills up over time (a month) but we restart the server to address. In other words, we manually address before we run out of memory. Since we never actually run out of memory, will ClassLoaderLeakPreventor help this issue? Thank you!

    • If you are doing redeploys that leak PermGen/Metaspace memory, then yes, this library is designed to help so you would no longer need to restart the server to avoid eventually gettingjava.lang.OutOfMemoryError.

      • Tim Clotworthy

        Thanks Mattias. I have implemented ClassLoaderLeakPreventor in a limited testing environment. I have an additional question. Is there any way to monitor or log the listener’s activity? For instance, whether it actually ever successfully identified a leak and successfully recovered the memory? I went to github site but did not see anything related to that. Otherwise how do you know that it is actually working (other than if you aren’t running out of memory any longer)? Thanks again!

        • The preventor does log any actions that are being taken, which it believes otherwise would have lead to a leak. (That being said, it is possible the leak would have been avoided for other reasons, such as the application server taking care of it or a known bug in a third party library having been fixed.) It is recommended to watch the logs and make corrections to your code and/or upgrade you third party libraries accordingly – or report bugs against them if issues is previously unknown.

          This library however is primarily meant for preventing the leaks, rather than detecting. For detection you may want to look at Arit or Plumbr. Or by all means follow the instruction in the first post in the series.

          • Tim Clotworthy

            Oh ok, so I guess you are saying that if the preventor took any actions, I should see them on catalina.out logging?

          • The library (2.x) uses java.util.logging out of the box, but this can be changed to log to stdout/stderr, if that better suits your setup.

            I’m not a Tomcat user myself, so you should check the documentation where Tomcat outputs by default.