Apologies if this topic has been discussed/resolved elsewhere. TBH I'm sure it has. If so, please just tell me to go search. But in any case the implications are big enough that it seems worth another discussion. Dominik Pospisil is running some perf tests and is seeing a roughly 10x degradation in the # of ejb3 sfsb requests/sec a cluster can handle w/ 5.1.0.Beta1 vs EAP 4.3. A thread dump on one of the nodes shows 300 threads contending for a BaseClassLoader object's monitor. The contention is through the loadClass(String, boolean) method, synchronized(this) statement at line 411. I believe the key issue here is we are not caching classes in BaseClassLoaderDomain. Following stack trace from the thread that currently holds the monitor illustrates: | "WorkerThread#189[10.16.88.183:37775]" prio=10 tid=0x66c81000 nid=0x5645 runnable [0x5c184000..0x5c185eb0] | java.lang.Thread.State: RUNNABLE | at java.lang.Throwable.fillInStackTrace(Native Method) | - locked <0xe74f8600> (a java.security.PrivilegedActionException) | at java.lang.Throwable.<init>(Throwable.java:241) | at java.lang.Exception.<init>(Exception.java:77) | at java.security.PrivilegedActionException.<init>(PrivilegedActionException.java:48) | at java.security.AccessController.doPrivileged(Native Method) | at java.net.URLClassLoader.findClass(URLClassLoader.java:188) | at java.lang.ClassLoader.loadClass(ClassLoader.java:307) | - locked <0x756a3fe8> (a sun.misc.Launcher$AppClassLoader) | at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) | - locked <0x756a3fe8> (a sun.misc.Launcher$AppClassLoader) | at java.lang.ClassLoader.loadClass(ClassLoader.java:300) | - locked <0x757cc198> (a org.jboss.system.NoAnnotationURLClassLoader) | at java.lang.ClassLoader.loadClass(ClassLoader.java:252) | at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) | - locked <0x757cc198> (a org.jboss.system.NoAnnotationURLClassLoader) | at java.lang.Class.forName0(Native Method) | at java.lang.Class.forName(Class.java:247) | at org.jboss.classloader.plugins.loader.ClassLoaderToLoaderAdapter.loadClass(ClassLoaderToLoaderAdapter.java:172) | at org.jboss.classloader.spi.ClassLoaderDomain.loadClassFromParent(ClassLoaderDomain.java:352) | at org.jboss.classloader.spi.ClassLoaderDomain.loadClassBefore(ClassLoaderDomain.java:307) | at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:246) | at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1102) | at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:772) | at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:415) | - locked <0x79d9a068> (a org.jboss.classloader.spi.base.BaseClassLoader) | at java.lang.ClassLoader.loadClass(ClassLoader.java:252) | at org.jboss.ejb3.metadata.annotation.AnnotationRepositoryToMetaData.loadClass(AnnotationRepositoryToMetaData.java:216) | at org.jboss.ejb3.metadata.annotation.AnnotationRepositoryToMetaData.hasAnnotation(AnnotationRepositoryToMetaData.java:328) | at org.jboss.aop.Advisor.hasAnnotation(Advisor.java:889) | at org.jboss.aop.Advisor.hasAnnotation(Advisor.java:865) | at org.jboss.ejb3.interceptors.aop.LifecycleCallbacks.createLifecycleCallbackInterceptors(LifecycleCallbacks.java:101) | at org.jboss.ejb3.EJBContainer.invokeCallback(EJBContainer.java:1112) | at org.jboss.ejb3.stateful.StatefulContainer.invokePrePassivate(StatefulContainer.java:668) | at org.jboss.ejb3.stateful.StatefulBeanContext.preReplicate(StatefulBeanContext.java:544) | at org.jboss.ejb3.cache.tree.StatefulTreeCache.putInCache(StatefulTreeCache.java:510) | at org.jboss.ejb3.cache.tree.StatefulTreeCache.create(StatefulTreeCache.java:123) | at org.jboss.ejb3.stateful.StatefulContainer.createSession(StatefulContainer.java:388) | at org.jboss.ejb3.session.SessionContainer.createSession(SessionContainer.java:677) | at org.jboss.ejb3.proxy.factory.session.stateful.StatefulSessionProxyFactoryBase.getNewSessionId(StatefulSessionProxyFactoryBase.java:296) | at org.jboss.ejb3.proxy.factory.session.stateful.StatefulSessionProxyFactoryBase.createProxyBusiness(StatefulSessionProxyFactoryBase.java:160) | at sun.reflect.GeneratedMethodAccessor304.invoke(Unknown Source) | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | at java.lang.reflect.Method.invoke(Method.java:597) | at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:121) | at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:908) | at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:742) | - locked <0x805023c8> (a org.jboss.remoting.transport.socket.ServerThread) | at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:695) | at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:549) | at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230) Here we have an EJB3 SFSB deployment, no loader repository configured so it is using the default classloading domain. The call stack is trying to load class javax.ejb.PrePassivate, which wasn't initialized by the EJB3 deployment's classloader, so the deployment's BaseClassLoader is asking the BaseClassLoadingDomain for the class. The class has already been loaded, by whatever classloader loaded common/lib/jboss-javaee.jar. This is an extremely common call pattern. AFAICT to return the already loaded class in EAP 4.3 vs 5.x we have, in simplified form: EAP 4.3 a single get from a concurrent hash map: return UnifiedLoaderRepository3.classes.get(name); AS 5.x 1) Ask the parent for the class, which calls ClassLoader.loadClass() proceeding all the way to the bootstrap classloader, executing privileged blocks throwing CNFEs wrapped inside PrivilegedActionExceptions at each level and then catching and discarding them. The above stack trace catches a thread in the middle of creating a PrivilegedActionException. 2) Executing BaseClassLoaderDomain.findLoader logic to find the Loader that loaded the class. 3) Scheduling and processing ClassLoadingTask to get the class from the Loader. Yikes! If the 4.x approach is too much of a "big ball of mud" can't we at least find a way to skip step #1 if the class was already loaded from one of the loaders associated with the domain? In forum thread at http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4218813#... pbmwg reports a similar problem, with a 66% performance improvement via eliminating just one source of calls to BaseClassLoader.loadClass(). View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4226629#... Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&a...