[jboss-jira] [JBoss JIRA] Closed: (JBREM-1127) ClassCastExceptions arising from Unmarsharller/Classloader caching in MicroRemoteClientInvoker

ClassCastExceptions arising from Unmarsharller/Classloader caching in MicroRemoteClientInvoker ---------------------------------------------------------------------------------------------- Key: JBREM-1127 URL: https://jira.jboss.org/jira/browse/JBREM-1127 Project: JBoss Remoting Issue Type: Bug Security Level: Public(Everyone can see) Affects Versions: 2.2.0.GA (Bluto), 2.2.0.SP1, 2.2.0.SP2, 2.2.0.SP3, 2.2.0.SP4, 2.2.1.GA, 2.2.2.GA, 2.2.2.GA_CP01, 2.2.2.SP1, 2.2.2.SP2, 2.2.2.GA_CP02, 2.2.2.GA_CP03, 2.2.2.SP4, 2.2.2.SP5, 2.2.2.SP7, 2.2.2.SP8, 2.2.2.SP9, 2.2.2.SP10, 2.2.2.SP11 Reporter: Galder Zamarreño Assignee: Ron Sigal Priority: Critical Fix For: 2.2.3 Attachments: jboss-remoting.jar, jboss-remoting.jar, jbrem1127.zip, patch.txt, server.log, server.logs.1905.zip, stacktrace-051409 Classloading when accessing a remote bean from within 2 separate isolated EARs in AS is broken. Let's say you have a bean, A, deployed in node2 and this bean is accessed from two isolated EARs in node1 and this bean returns a instance of Foo. If you run paralell requests to these two separate EARs, you'll randomly get ClassCastExceptions. This is due to these two isolated EARs could potentially be using the same MicroRemoteClientInvoker instance, and hence the same unmarshaller. So, you could easily get this situation: 1. Invocation for bean A is proxied through isolated EAR-1 in node1. A new client invoker is created. 2009-05-04 16:31:04,489 31905 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] (WorkerThread#0[127.0.0.2:50932]:) SocketClientInvoker[5e5f92, socket://lo3:3873] constructed ... // new logged line below server.log:12767:2009-05-04 16:31:04,489 31905 TRACE [org.jboss.remoting.MicroRemoteClientInvoker] (WorkerThread#0[127.0.0.2:50932]:) New unmarshaller set to org.jboss.remoting.marshal.serializable.SerializableUnMarshaller@2606b8 with classloader org.jboss.remoting.loading.RemotingClassLoader@10f3801 2. Loads Foo (or in this case ResponseMsg from EAR-1): 2009-05-04 16:31:04,534 31950 TRACE [org.jboss.remoting.loading.ObjectInputStreamWithClassLoader] (WorkerThread#0[127.0.0.2:50932]:) Calling Class.forName(classcastexceptiontest.process.ResponseMsg, false, org.jboss.remoting.loading.RemotingClassLoader@10f3801) 2009-05-04 16:31:04,535 31951 TRACE [org.jboss.remoting.loading.ObjectInputStreamWithClassLoader] (WorkerThread#0[127.0.0.2:50932]:) Resolved class classcastexceptiontest.process.ResponseMsg.ClassLoader=org.jboss.mx.loading.UnifiedClassLoader3@1d4f279{ url=file:/home/galder/jboss/code/eap/4.3.0.ga.cp04/build/output/jboss-4.3.0.GA_CP04/server/290590_1/tmp/deploy/tmp4913079009040294654processinvoker1.ear ,addedOrder=37} ..org.jboss.mx.loading.UnifiedClassLoader3@1d4f279{ url=file:/home/galder/jboss/code/eap/4.3.0.ga.cp04/build/output/jboss-4.3.0.GA_CP04/server/290590_1/tmp/deploy/tmp4913079009040294654processinvoker1.ear ,addedOrder=37} ..org.jboss.mx.loading.HeirarchicalLoaderRepository3$NoParentClassLoader@1178281 ..org.jboss.system.server.NoAnnotationURLClassLoader@1e4457d ..sun.misc.Launcher$AppClassLoader@17182c1 ....file:/home/galder/jboss/code/eap/4.3.0.ga.cp04/build/output/jboss-4.3.0.GA_CP04/bin/run.jar ....file:/usr/java/jdk1.6.0_12/lib/tools.jar ..sun.misc.Launcher$ExtClassLoader@13f5d07 ....file:/usr/java/jdk1.6.0_12/jre/lib/ext/dnsns.jar ....file:/usr/java/jdk1.6.0_12/jre/lib/ext/sunjce_provider.jar ....file:/usr/java/jdk1.6.0_12/jre/lib/ext/localedata.jar ....file:/usr/java/jdk1.6.0_12/jre/lib/ext/sunpkcs11.jar ++++CodeSource: (file:/home/galder/jboss/code/eap/4.3.0.ga.cp04/build/output/jboss-4.3.0.GA_CP04/server/290590_1/tmp/deploy/tmp4913079009040294654processinvoker1.ear-contents/process-intf.jar <no signer certificates>) Implemented Interfaces: ++interface java.io.Serializable ++++ClassLoader: null ++++Null CodeSource 2. Invocation for bean B is proxied through isolated EAR-2 in node1. A cached invoker is used: 2009-05-04 16:31:04,516 31932 TRACE [org.jboss.remoting.InvokerRegistry] (WorkerThread#1[127.0.0.2:50931]:) Found and returning cached client invoker (SocketClientInvoker[5e5f92, socket://lo3:3873]) And the unrmarshaller is already set with the wrong classloader!!: 2009-05-04 16:31:04,516 31932 TRACE [org.jboss.remoting.MicroRemoteClientInvoker] (WorkerThread#1[127.0.0.2:50931]:) Unmarshaller is already set to org.jboss.remoting.marshal.serializable.SerializableUnMarshaller@2606b8 3. So, with the wrong classloader, it loads Foo (or ResponseMsg) from EAR-1 again when it should have loaded it from EAR-2: 2009-05-04 16:31:04,524 31940 TRACE [org.jboss.remoting.loading.ObjectInputStreamWithClassLoader] (WorkerThread#1[127.0.0.2:50931]:) Resolved class classcastexceptiontest.process.ResponseMsg.ClassLoader=org.jboss.mx.loading.UnifiedClassLoader3@1d4f279{ url=file:/home/galder/jboss/code/eap/4.3.0.ga.cp04/build/output/jboss-4.3.0.GA_CP04/server/290590_1/tmp/deploy/tmp4913079009040294654processinvoker1.ear ,addedOrder=37} ..org.jboss.mx.loading.UnifiedClassLoader3@1d4f279{ url=file:/home/galder/jboss/code/eap/4.3.0.ga.cp04/build/output/jboss-4.3.0.GA_CP04/server/290590_1/tmp/deploy/tmp4913079009040294654processinvoker1.ear ,addedOrder=37} ..org.jboss.mx.loading.HeirarchicalLoaderRepository3$NoParentClassLoader@1178281 ..org.jboss.system.server.NoAnnotationURLClassLoader@1e4457d ..sun.misc.Launcher$AppClassLoader@17182c1 ....file:/home/galder/jboss/code/eap/4.3.0.ga.cp04/build/output/jboss-4.3.0.GA_CP04/bin/run.jar ....file:/usr/java/jdk1.6.0_12/lib/tools.jar ..sun.misc.Launcher$ExtClassLoader@13f5d07 ....file:/usr/java/jdk1.6.0_12/jre/lib/ext/dnsns.jar ....file:/usr/java/jdk1.6.0_12/jre/lib/ext/sunjce_provider.jar ....file:/usr/java/jdk1.6.0_12/jre/lib/ext/localedata.jar ....file:/usr/java/jdk1.6.0_12/jre/lib/ext/sunpkcs11.jar ++++CodeSource: (file:/home/galder/jboss/code/eap/4.3.0.ga.cp04/build/output/jboss-4.3.0.GA_CP04/server/290590_1/tmp/deploy/tmp4913079009040294654processinvoker1.ear-contents/process-intf.jar <no signer certificates>) Implemented Interfaces: ++interface java.io.Serializable ++++ClassLoader: null ++++Null CodeSource 4. Result is a CCE: 2009-05-04 16:31:04,525 31941 ERROR [classcastexceptiontest.processinvoker.ProcessInvoker2] (WorkerThread#1[127.0.0.2:50931]:) Caught exception java.lang.ClassCastException: classcastexceptiontest.process.ResponseMsg cannot be cast to classcastexceptiontest.process.ResponseMsg at $Proxy74.doProcess(Unknown Source) at classcastexceptiontest.processinvoker.ProcessInvoker2.invokeProcess(ProcessInvoker2.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:57) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:94) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:70) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:108) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:280) at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106) at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:866) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173) How to resolve this? At first glance, the most trivial thing to do is to have an unmarshaller that simply uses whatever Thread.currentThread().getContextClassLoader() is set to, rather than being caching unmarshaller and classloaders.