[jboss-user] [JBoss Messaging] - Periodic org.jboss.aop.NotFoundInDispatcherException prevent

We are running JBoss Messaging 1.0.1SP4. After some period of use (seems to happen after many messages have been sent through the system, but not necessarily related to load), we see the messages below in the logs, and all further sends seem to fail until the server is restarted. Once the server is restarted, the client automatically reconnects, and everything works again. Server Log of org.jboss.messaging and org.jboss.remoting at debug level: | Removed org.jboss.remoting.transport.local.LocalClientInvoker@686ecb from registry | Using local client invoker for invocation. | ServerInvoker (SocketServerInvoker[172.31.40.54:2195]) removing client callback handler with session id of 4sv141i-vyg6ek-eyta2mrp-1-ez07tnm4-2wh+4sv141i-vyg6ek-eyta2mrp-1-ez07tnm4-2wi. | removed org.jboss.remoting.transport.local.LocalClientInvoker@4201d9 from registry | decremented org.jboss.remoting.transport.local.LocalClientInvoker@e6c243's count, current count 12 | Server connection lost to client (session id = 4sv141k-a5vaz8-ez00omj4-1-ez00oo3z-4 | SocketServerInvoker[0.0.0.0:4457].invoke() call failed: Object with oid: -2147481983 was not found in the Dispatcher | SocketServerInvoker[0.0.0.0:4457].invoke() call failed | org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147481983 was not found in the Dispatcher | at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85) | at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715) | at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552) | at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377) | at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159) | SocketServerInvoker[0.0.0.0:4457].invoke() call failed: Object with oid: -2147481983 was not found in the Dispatcher | SocketServerInvoker[0.0.0.0:4457].invoke() call failed | org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147481983 was not found in the Dispatcher | at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85) | at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715) | at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552) | at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377) | at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159) | ... | Client Log of org.jboss.messaging at debug level: | describeObject for class=org.jboss.remoting.InvocationRequest | describeObject::a new reference 1 | defaultWrite::org.jboss.remoting.InvocationRequest contains 1 slots | defaultWrite:: slot 0 NR=org.jboss.remoting.InvocationRequest from parentClass=org.jboss.remoting.InvocationRequest | writeSlotWithFields slot=org.jboss.remoting.InvocationRequest and 6 fields | writeSlotWithFields FieldNr=0 | writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField sessionId with object=java.lang.String | writeSlotWithFields FieldNr=1 | writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField subsystem with object=java.lang.String | writeSlotWithFields FieldNr=2 | writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField arg with object=org.jboss.jms.server.remoting.MessagingMarshallable | describeObject for class=org.jboss.jms.server.remoting.MessagingMarshallable | describeObject::a new reference 5 | defaultWrite::org.jboss.jms.server.remoting.MessagingMarshallable contains 1 slots | defaultWrite:: slot 0 NR=org.jboss.jms.server.remoting.MessagingMarshallable from parentClass=org.jboss.jms.server.remoting.MessagingMarshallable | writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable and 2 fields | writeSlotWithFields FieldNr=0 | writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable primitiveField version with object=NULL | writeSlotWithFields FieldNr=1 | writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable objectField load with object=org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442 | describeObject for class=org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442 | describeObject::a new reference 7 | describeObject for class=org.jboss.aop.metadata.SimpleMetaData | describeObject::a new reference 9 | writeSlotWithFields FieldNr=3 | writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField requestPayload with object=NULL | writeSlotWithFields FieldNr=4 | writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField returnPayload with object=NULL | writeSlotWithFields FieldNr=5 | writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField locator with object=NULL | objectFromDescription::reading new definition | Reading string with utfSize=37 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | hash on field org.jboss.remoting.InvocationResponse = -5123635159962572581 | lookupMethodOnHierarchy::class=org.jboss.remoting.InvocationResponse looking for readResolve | lookupMethodOnHierarchy::currentClass=class org.jboss.remoting.InvocationResponse | lookupMethodOnHierarchy::class=org.jboss.remoting.InvocationResponse looking for writeReplace | lookupMethodOnHierarchy::currentClass=class org.jboss.remoting.InvocationResponse | hash on field org.jboss.remoting.InvocationResponse = -5123635159962572581 | hash on field java.lang.String$sessionId = 7208879560699038148 | hash on field boolean$isException = 6944741943427072418 | hash on field java.lang.Object$result = -1910178885101376583 | hash on field java.util.Map$payload = -311608580347357596 | Reading object for id=1 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = org.jboss.remoting.InvocationResponse | defaultRead::class org.jboss.remoting.InvocationResponse contains 1 slots | defaultRead::slot[0]=org.jboss.remoting.InvocationResponse | readSlotWithFields slot=org.jboss.remoting.InvocationResponse | FieldName on Read=sessionId | readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=sessionId<<-reading Object | Reading string with utfSize=36 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=isException | FieldName on Read=result | readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=result<<-reading Object | objectFromDescription::reading new definition | Reading string with utfSize=43 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | hash on field org.jboss.aop.NotFoundInDispatcherException = -5393238603831683671 | lookupMethodOnHierarchy::class=org.jboss.aop.NotFoundInDispatcherException looking for readResolve | lookupMethodOnHierarchy::currentClass=class org.jboss.aop.NotFoundInDispatcherException | lookupMethodOnHierarchy::currentClass=class org.jboss.aop.DispatcherConnectException | lookupMethodOnHierarchy::currentClass=class java.lang.Exception | lookupMethodOnHierarchy::currentClass=class java.lang.Throwable | lookupMethodOnHierarchy::class=org.jboss.aop.NotFoundInDispatcherException looking for writeReplace | lookupMethodOnHierarchy::currentClass=class org.jboss.aop.NotFoundInDispatcherException | lookupMethodOnHierarchy::currentClass=class org.jboss.aop.DispatcherConnectException | lookupMethodOnHierarchy::currentClass=class java.lang.Exception | lookupMethodOnHierarchy::currentClass=class java.lang.Throwable | hash on field org.jboss.aop.NotFoundInDispatcherException = -5393238603831683671 | hash on field java.lang.Object$oid = 5891181997928938037 | hash on field org.jboss.aop.DispatcherConnectException = -2548359991833714121 | hash on field java.lang.Exception = 7600092213321646919 | hash on field java.lang.Throwable = -7779525601374451073 | hash on field java.lang.String$detailMessage = -4333528023742127755 | hash on field java.lang.Throwable$cause = 4080765070623846565 | hash on field [Ljava.lang.StackTraceElement;$stackTrace = -4842200784206624673 | Reading object for id=4 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = org.jboss.aop.NotFoundInDispatcherException | defaultRead::class org.jboss.aop.NotFoundInDispatcherException contains 4 slots | defaultRead::slot[0]=java.lang.Throwable | readSlotWithFields slot=java.lang.Throwable | FieldName on Read=detailMessage | readSlotWithFields slot=java.lang.Throwable field=detailMessage<<-reading Object | Reading string with utfSize=60 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=cause | readSlotWithFields slot=java.lang.Throwable field=cause<<-reading Object | objectFromDescription::reading circular definition reference=4 | FieldName on Read=stackTrace | readSlotWithFields slot=java.lang.Throwable field=stackTrace<<-reading Object | objectFromDescription::reading new definition | Reading string with utfSize=30 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | hash on field [Ljava.lang.StackTraceElement; = -8287581384409355425 | lookupMethodOnHierarchy::class=[Ljava.lang.StackTraceElement; looking for readResolve | lookupMethodOnHierarchy::currentClass=class [Ljava.lang.StackTraceElement; | lookupMethodOnHierarchy::class=[Ljava.lang.StackTraceElement; looking for writeReplace | lookupMethodOnHierarchy::currentClass=class [Ljava.lang.StackTraceElement; | hash on field [Ljava.lang.StackTraceElement; = -8287581384409355425 | Reading object for id=7 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = [Ljava.lang.StackTraceElement; | objectFromDescription::reading new definition | Reading string with utfSize=27 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | hash on field java.lang.StackTraceElement = -5341604412408429699 | lookupMethodOnHierarchy::class=java.lang.StackTraceElement looking for readResolve | lookupMethodOnHierarchy::currentClass=class java.lang.StackTraceElement | lookupMethodOnHierarchy::class=java.lang.StackTraceElement looking for writeReplace | lookupMethodOnHierarchy::currentClass=class java.lang.StackTraceElement | hash on field java.lang.StackTraceElement = -5341604412408429699 | hash on field java.lang.String$declaringClass = -4020810388859008069 | hash on field java.lang.String$methodName = 2043831539205426629 | hash on field java.lang.String$fileName = -3886410644765885867 | hash on field int$lineNumber = 1246520928412557129 | Reading object for id=9 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement | defaultRead::class java.lang.StackTraceElement contains 1 slots | defaultRead::slot[0]=java.lang.StackTraceElement | readSlotWithFields slot=java.lang.StackTraceElement | FieldName on Read=declaringClass | readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object | Reading string with utfSize=24 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=methodName | readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object | Reading string with utfSize=6 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=fileName | readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object | Reading string with utfSize=15 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=lineNumber | objectFromDescription::reading new definition | Reading object for id=14 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement | defaultRead::class java.lang.StackTraceElement contains 1 slots | defaultRead::slot[0]=java.lang.StackTraceElement | readSlotWithFields slot=java.lang.StackTraceElement | FieldName on Read=declaringClass | readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object | Reading string with utfSize=56 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=methodName | readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object | FieldName on Read=fileName | readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object | Reading string with utfSize=31 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=lineNumber | objectFromDescription::reading new definition | Reading object for id=17 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement | defaultRead::class java.lang.StackTraceElement contains 1 slots | defaultRead::slot[0]=java.lang.StackTraceElement | readSlotWithFields slot=java.lang.StackTraceElement | FieldName on Read=declaringClass | readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object | Reading string with utfSize=32 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=methodName | readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object | FieldName on Read=fileName | readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object | Reading string with utfSize=18 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=lineNumber | objectFromDescription::reading new definition | Reading object for id=20 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement | defaultRead::class java.lang.StackTraceElement contains 1 slots | defaultRead::slot[0]=java.lang.StackTraceElement | readSlotWithFields slot=java.lang.StackTraceElement | FieldName on Read=declaringClass | readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object | Reading string with utfSize=48 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=methodName | readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object | Reading string with utfSize=17 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=fileName | readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object | Reading string with utfSize=17 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=lineNumber | objectFromDescription::reading new definition | Reading object for id=24 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement | defaultRead::class java.lang.StackTraceElement contains 1 slots | defaultRead::slot[0]=java.lang.StackTraceElement | readSlotWithFields slot=java.lang.StackTraceElement | FieldName on Read=declaringClass | readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object | FieldName on Read=methodName | readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object | Reading string with utfSize=5 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=fileName | readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object | FieldName on Read=lineNumber | objectFromDescription::reading new definition | Reading object for id=26 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement | defaultRead::class java.lang.StackTraceElement contains 1 slots | defaultRead::slot[0]=java.lang.StackTraceElement | readSlotWithFields slot=java.lang.StackTraceElement | FieldName on Read=declaringClass | readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object | FieldName on Read=methodName | readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object | Reading string with utfSize=3 isLong=false | readString::pulling data to Buffer at pos 0 size= 0 | FieldName on Read=fileName | readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object | FieldName on Read=lineNumber | defaultRead::slot[1]=java.lang.Exception | readSlotWithFields slot=java.lang.Exception | defaultRead::slot[2]=org.jboss.aop.DispatcherConnectException | readSlotWithFields slot=org.jboss.aop.DispatcherConnectException | defaultRead::slot[3]=org.jboss.aop.NotFoundInDispatcherException | readSlotWithFields slot=org.jboss.aop.NotFoundInDispatcherException | FieldName on Read=oid | readSlotWithFields slot=org.jboss.aop.NotFoundInDispatcherException field=oid<<-reading Object | FieldName on Read=payload | readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=payload<<-reading Object | Caught Exception: | org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483579 was not found in the Dispatcher | at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85) | at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715) | at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552) | at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377) | at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159) | at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163) | at org.jboss.remoting.Client.invoke(Client.java:1414) | at org.jboss.remoting.Client.invoke(Client.java:511) | at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111) | at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) | at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117) | at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) | at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) | at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) | at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) | at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) | at org.jboss.jms.client.delegate.ClientConnectionDelegate.stop(ClientConnectionDelegate.java) | at org.jboss.jms.client.JBossConnection.stop(JBossConnection.java:126) | at oracle.tip.adapter.jms.JMS.JMSConnection.stop(JMSConnection.java:432) | at oracle.tip.adapter.jms.JMS.JMSMessageConsumer.stop(JMSMessageConsumer.java:252) | at oracle.tip.adapter.jms.inbound.JmsConsumer.stop(JmsConsumer.java:164) | at oracle.tip.adapter.jms.JmsEndpoint.stop(JmsEndpoint.java:150) | at oracle.tip.adapter.jms.JmsResourceAdapter.endpointDeactivation(JmsResourceAdapter.java:196) | at oracle.tip.adapter.fw.jca.AdapterFrameworkImpl.endpointDeactivation(AdapterFrameworkImpl.java:590) | at oracle.tip.adapter.fw.agent.jca.JCAActivationAgent.deactivateInboundJcaEndpoint(JCAActivationAgent.java:953) | at oracle.tip.adapter.fw.agent.jca.JCAActivationAgent.uninit(JCAActivationAgent.java:458) | at com.collaxa.cube.engine.core.BaseCubeProcess.disableActivationAgents(BaseCubeProcess.java:516) | at com.collaxa.cube.engine.core.BaseCubeProcess.uninit(BaseCubeProcess.java:321) | at com.collaxa.cube.engine.deployment.CubeProcessLoader.destroy(CubeProcessLoader.java:589) | at com.collaxa.cube.engine.deployment.CubeProcessHolder.uninit(CubeProcessHolder.java:141) | at com.collaxa.cube.engine.CubeEngine.uninit(CubeEngine.java:794) | at com.collaxa.cube.admin.ServerManager.uninit(ServerManager.java:334) | at com.collaxa.cube.admin.ServerManager$1.run(ServerManager.java:150) | Any ideas on where to go from here to get some possible idea what may be causing this error? View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4026006#... Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...