[jboss-user] [Performance Tuning] - ClassLoader bottleneck in 5.1.0.GA

[Mikal Henriksen]

Mikal Henriksen [https://community.jboss.org/people/stylpe] created the discussion

"ClassLoader bottleneck in 5.1.0.GA"

To view the discussion, visit: https://community.jboss.org/message/831673#831673

--------------------------------------------------------------
So it looks like our app is affected by JBCL-185 and/or JBCL-100. We were running visualvm during load testing, and discovered that EJB proxy instances had a very high self-time. This is a SLSB with a @Local interface, so it should be pretty fast.

We did a thread dump, and it turns out most of the servlet threads were blocked waiting for a class loader. Some examples:

Several are like this: 
Thread: ajp-0.0.0.0-8009-11 : priority:5, demon:true, threadId:198, threadState:BLOCKED
- waiting on <0x22d22fe3> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)
java.lang.ClassLoader.loadClass(ClassLoader.java:291)
java.lang.ClassLoader.loadClass(ClassLoader.java:295)
- locked <0x2c25bf62> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)
java.lang.ClassLoader.loadClass(ClassLoader.java:247)
org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)
org.jboss.ejb3.common.lang.SerializableMethod.getClassType(SerializableMethod.java:282)
org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:233)
org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:220)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.handleInvocationDirectly(SessionProxyInvocationHandlerBase.java:240)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:182)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
$Proxy815.getAdaptedItem(Unknown Source)
[servlet method here]

The thread owning 0x22d22fe3:

Thread: ajp-0.0.0.0-8009-18 : priority:5, demon:true, threadId:205, threadState:BLOCKED
- waiting on <0x1b4ddacc> (a org.jboss.classloader.spi.base.BaseClassLoader)
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:247)
org.jboss.classloader.plugins.loader.ClassLoaderToLoaderAdapter.loadClass(ClassLoaderToLoaderAdapter.java:172)
org.jboss.classloader.spi.ClassLoaderDomain.loadClassFromParent(ClassLoaderDomain.java:352)
org.jboss.classloader.spi.ClassLoaderDomain.loadClassAfter(ClassLoaderDomain.java:323)
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:275)
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
- locked <0x357d9553> (a org.jboss.classloader.spi.base.BaseClassLoader)
java.lang.ClassLoader.loadClass(ClassLoader.java:295)
- locked <0x22d22fe3> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)
java.lang.ClassLoader.loadClass(ClassLoader.java:295)
- locked <0x7866dc25> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)
java.lang.ClassLoader.loadClass(ClassLoader.java:247)
org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)
org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:239)
org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:220)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:182)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
$Proxy815.getAdaptedItem(Unknown Source)
[servlet method here]

The thread owning 0x1b4ddacc:

Thread: ajp-0.0.0.0-8009-5 : priority:5, demon:true, threadId:192, threadState:RUNNABLE
java.util.Arrays.copyOfRange(Arrays.java:3209)
java.lang.String.(String.java:215)
java.lang.StringBuilder.toString(StringBuilder.java:430)
org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:448)
- locked <0x1b4ddacc> (a org.jboss.classloader.spi.base.BaseClassLoader)
java.lang.ClassLoader.loadClass(ClassLoader.java:295)
- locked <0x24c3e995> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)
java.lang.ClassLoader.loadClass(ClassLoader.java:247)
org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)
org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:239)
org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:135)
org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:216)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
$Proxy815.getAdaptedItem(Unknown Source)
[servlet method here]


According to what I've read on the issue pages and in discussions here, the root cause is the classes aren't properly cached, and thus have to be loaded on every single session bean invocation.

Is there a way to work around this issue? Upgrading JBoss is not an option, but if upgrading only the class loader component is possible, we could do that.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/831673#831673]

Start a new discussion in Performance Tuning at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2078]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20130806/b6a69f14/attachment-0001.html