Classloader leaks II – Find and work around unwanted references

If you just want a quick fix to the problem without understanding the theory, jump to part IV introducing the ClassLoader Leak Prevention library.

In my previous post we learnt how to locate classloader leaks using Eclipse Memory Analyzer (MAT).

This time we will discuss different reasons for leaks, look at an example of a leak in a third party library, and see how we can fix that leak by a workaround.

Different reasons for ClassLoader leaks

In order to know what you should be looking for in your heapdump analysis, we could categorize ClassLoader leaks into three different types. In the end, they are all just variants of the first one.

  1. References from outside your webapp – that is from the application server or the JDK classes – to either the ClassLoader itself or one of the classes it has loaded (which in turn has a reference to the ClassLoader), including any instances of such classes.
     
  2. Threads running inside your webapp. If you spawn new threads from within your web application that may not terminate, they are likely to prevent your ClassLoader from being garbage collected. This can happen even if the thread does not use any of the classes loaded by your webapps ClassLoader. This is because threads have a context classloader, to which there is a reference (contextClassLoader) in the java.lang.Thread class. More about this in the next post.
     
  3. ThreadLocals with values whose class is loaded in your webapp. If you use ThreadLocals in your webapp, you need to explicitly clear all ThreadLocals before the webapp closes down. This is because a) the application server uses a thread pool, which means that the thread will outlive your webapp instance and b) ThreadLocal values are actually stored in the java.lang.Thread object. Therefore, this is just a variation of 1.
    (Note: This may be the case most likely created by yourself, but also exists in third party libraries.)

Example of reference from outside your application

When trying to hunt down a ClassLoader leak in our web application, I created a little JSP page in which I looped through all the third party JARs of our application. I tried to load every single class that was found in a custom ClassLoader, added a ZombieMarker to the ClassLoader (see previous post) and then disposed the ClassLoader. I ran the JSP page over and over again until I got a java.lang.OutOfMemoryError: PermGen space. That is, I was able to trigger ClassLoader leaks just by loading classes from our third party libraries… 🙁 It actually turned out to be more than one of them, that triggered this behaviour.

Here is a MAT trace for one of them:

(In this picture, it’s not obvious where our ClassLoader is. The custom ClassLoader was an anonymous inner class in my JSP, so it’s the second entry with the strange class name ending with $1.)

At first glance, it may seem like this is type 2 above, with a running thread. This is not the case however, since the thread itself is not the GC root (not at the bottom level). In fact, there is a Thread involved, but it is not running.

Rather we can see that what keeps our ClassLoader from being garbage collected is a reference from outside the webapp (java.lang.*) to an instance of com.sun.media.jai.codec.TempFileCleanupThread, which in turn is loaded by our ClassLoader. From the names of the referenced and referencing (java.lang.ApplicationShutdownHook) classes, I suspected that a JVM shutdown hook was added by some Java Advanced Imaging (JAI) class when it was loaded.

The com.sun.media.jai.codec.TempFileCleanupThread class is in the Codec part of JAI; version 1.1.2_01 in our case. The sources can be found in the official SVN repo (1.1.2_01 tag). As you can see, TempFileCleanupThread.java class is not in that list. That is because someone thought is was a great idea to put it as a package protected class in FileCacheSeekableStream.java.

There we can also find the source of the leak.

    // Create the cleanup thread. Use reflection to preserve compile-time
    // compatibility with JDK 1.2.
    static {
        try {
            Method shutdownMethod =
                Runtime.class.getDeclaredMethod("addShutdownHook",
                                                new Class[] {Thread.class});

            cleanupThread = new TempFileCleanupThread();

            shutdownMethod.invoke(Runtime.getRuntime(),
                                  new Object[] {cleanupThread});
        } catch(Exception e) {
            // Reset the Thread to null if Method.invoke failed.
            cleanupThread = null;
        }
    }

As suspected, there is a static block that (via reflection) adds a JVM shutdown hook, as soon as the com.sun.media.jai.codec.FileCacheSeekableStream class is loaded. Not very practical in a web application environment, since the JVM will will not shutdown until the application server is shut down.

The JAI TempFileCleanupThread is supposed to delete temporary files when the JVM shuts down. In a web application, what we want is probably to remove those temporary files as soon as the web application is redeployed. If this was our own code, we should have changed this. In this case it’s a third party library, and judging from the SVN trunk, this still has not been fixed, so upgrading doesn’t help. (This has been reported here.)

Cleaning up leaking references at redeploy

In order to clean up references as part of web application shutdown, to prevent ClassLoader leaks, there are two approaches. You can either put the code in the destroy() method of a Servlet that is load-on-startup

  <servlet servlet-name='cleanup' servlet-class='my.CleanupServlet'>
    <load-on-startup>1</load-on-startup>
  </servlet>

or (probably slightly more correct) you can create a javax.servlet.ServletContextListener and add the cleanup to the contextDestroyed() method.

  <listener>
    <listener-class>my.CleanupListener</listener-class>
  </listener>

The workaround

Fortunately, FileCacheSeekableStream keeps a reference to the shutdown hook in our case.

public final class FileCacheSeekableStream extends SeekableStream {

    /** A thread to clean up all temporary files on VM exit (VM 1.3+) */
    private static TempFileCleanupThread cleanupThread = null;

So let’s grab that reference and remove the shutdown hook. But we probably don’t just want to throw away the hook, since in theory that may leave us with temporary files that should has been deleted at JVM shutdown. Instead get the hook, remove it, and then run it immediately.

We may actually turn this into a generic method, to be reused for other third party shutdown hooks we want to remove. (System.out is used for logging, since logging frameworks usually needs to be cleaned up too, and I suggest you do that before calling this method.)

private static void removeShutdownHook(Class clazz, String field) {
  // Note that loading the class may add the hook if not yet present... 
  try {
    // Get the hook
    final Field cleanupThreadField = clazz.getDeclaredField(field);
    cleanupThreadField.setAccessible(true);
    Thread cleanupThread = (Thread) cleanupThreadField.get(null);

    if(cleanupThread != null) {
      // Remove hook to avoid PermGen leak
      System.out.println("  Removing " + cleanupThreadField + " shutdown hook");
      Runtime.getRuntime().removeShutdownHook(cleanupThread);
      
      // Run cleanup immediately
      System.out.println("  Running " + cleanupThreadField + " shutdown hook");
      cleanupThread.start();
      cleanupThread.join(60 * 1000); // Wait up to 1 minute for thread to run
      if(cleanupThread.isAlive())
        System.out.println("STILL RUNNING!!!");
      else
        System.out.println("Done");
    }
    else
      System.out.println("  No " + cleanupThreadField + " shutdown hook");
    
  }
  catch (NoSuchFieldException ex) {
    System.err.println("*** " + clazz.getName() + '.' + field + 
      " not found; has JAR been updated??? ***");
    ex.printStackTrace();
  }
  catch(Exception ex) {
    System.err.println("Unable to unregister " + clazz.getName() + '.' + field);
    ex.printStackTrace();
  }    
}

Now we just call that method in our application shutdown (CleanupServlet.destroy() / CleanupListener.contextDestroyed()) like so:

removeShutdownHook(com.sun.media.jai.codec.FileCacheSeekableStream.class,
  "cleanupThread");

In a worst case scenario, if there is no reference kept to the shutdown hook, we may use reflection into the JVM classes. It would look like this:

final Field field = 
  Class.forName("java.lang.ApplicationShutdownHooks").getDeclaredField("hooks");
field.setAccessible(true);
Map<Thread, Thread> shutdownHooks = (Map<Thread, Thread>) field.get(null);
// Iterate copy to avoid ConcurrentModificationException
for(Thread t : new ArrayList<Thread>(shutdownHooks.keySet())) {
  if(t.getClass().getName().equals("class.name.of.ShutdownHook")) { // TODO: Set name
    // Make sure it's from this web app instance
    if(t.getClass().getClassLoader().equals(this.getClass().getClassLoader())) {
      Runtime.getRuntime().removeShutdownHook(t); // Remove hook to avoid PermGen leak
      t.start(); // Run cleanup immediately
      t.join(60 * 1000); // Wait up to 1 minute for thread to run
    }
  }
}

That’s all for this post. Next time we’ll look at threads running within your ClassLoader.

Update – Bean Validation API begs “FIXME”

I can’t help but post an additional example, that I found just the other day. Had some PermGen errors in a new webapp and this is what I found:

Looking at Validation.java and the inner class javax.validation.Validation.DefaultValidationProviderResolver it does, at least in the current revision, contain these lines of code:

		//cache per classloader for an appropriate discovery
		//keep them in a weak hashmap to avoid memory leaks and allow proper hot redeployment
		//TODO use a WeakConcurrentHashMap
		//FIXME The List<VP> does keep a strong reference to the key ClassLoader, use the same model as JPA CachingPersistenceProviderResolver
		private static final Map<ClassLoader, List<ValidationProvider<?>>> providersPerClassloader =
				new WeakHashMap<ClassLoader, List<ValidationProvider<?>>>();

Isn’t that nice? In the Bean Validation API (JSR 303) – not an implementation but the API – there is a cache that have been created with hot redeployment in mind, and still it has the potential to leaks classloaders. Not only that – the authors of the code have been aware that it can leak classloaders, and still validation-api-1.0.0.GA.jar was released, without any means of manually telling the cache to release our ClassLoader. Sigh…

The leak is triggered when the API is shipped with your application server, but the implementation (Hibernate Validator in my case) is provided in your web application, and thus loaded with your classloader.

Using reflection like above, we stop the leak by getting hold of the Map and remove() our classloader. Alternatively, we could add the JAR of our Validation provider on the Application Server level, so that the cache will not reference our webapp ClassLoader at all.

Links to all parts in the series

Part I – How to find classloader leaks with Eclipse Memory Analyser (MAT)

Part II – Find and work around unwanted references

Part III – “Die Thread, die!”

Part IV – ThreadLocal dangers and why ThreadGlobal may have been a more appropriate name

Part V – Common mistakes and Known offenders

Part VI – “This means war!” (leak prevention library)

Presentation on Classloader leaks (video and slides)

  • Hal Deadman

    It sounds like you are getting ready to create a library full of code that cleans up known memory leaks? That would be great. I was wondering if you have seen this project: http://code.google.com/p/arit/. I haven’t had a chance to run it yet but it is focused on finding classloader memory leaks.

    • Hi Hal. I was not aware of Arit before – seems interesting.nnMy project however is not about detecting leaks, but about preventing them. Somewhat like what is included in Tomcat, although I will cover some leaks that they don’t – both generic and specific – plus this will be application server independent. The idea is that you add this component to your web application and then you should be “safe”.nnMost of the basic functionality is in place, and it seems to be working as planned :-)nI haven’t decided how finished it needs to be before I announce it.

      • Hal Deadman

        I just took a look at your ClasLoaderLeakPreventor. It’s just what I needed to be able to do Weblogic zero downtime deployments without worrying about running out of memory. I will definately try it out, possibly in about 30 applications. The single listener implementation is quite a nice contrast to Arit which is made up 36 modules and when I tried it on Weblogic it didn’t work (although they don’t try to support it). Thanks for designing it to be extended. I can’t believe you are looping through all the threads in the JVM and removing thread locals. Impressive.

        • I’m quite pleased with the ThreadLocal handling. In Tomcat they removed the redeploy clearing, and resorted to warning only, since they tried to actually remove the entry from the map/array, which could cause concurrency issues. I think they may have put clearing in the thread pool instead.nnSo first I also had a filter that could clear known offenders, or loop and clear all, on each request, which could affect performance.nBut my shutdown strategy of simply leaving the entries but “empty” them – to make them stale and later expunged by the JRE – seems to work so good I removed the filter part.nnFeel free to let me know how the deployment goes.

  • ao

    this is amazing.