[jboss-jira] [JBoss JIRA] (JBMESSAGING-1957) JBoss EAP5.2 clusterred nodes are giving error : java.lang.IllegalStateException: Don't know about node id:3..... and java.lang.IllegalStateException: Failed to remove binding, and..... java.lang.IllegalStateException: Cannot find object in dispatcher

JBoss EAP5.2 clusterred nodes are giving error : java.lang.IllegalStateException: Don't know about node id:3..... and java.lang.IllegalStateException: Failed to remove binding, and..... java.lang.IllegalStateException: Cannot find object in dispatcher ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Key: JBMESSAGING-1957 URL: https://issues.jboss.org/browse/JBMESSAGING-1957 Project: JBoss Messaging Issue Type: Feature Request Components: JMS Clustering, Messaging Core Distributed Support Environment: JBoss 5.2 Reporter: Manish Kushwaha Priority: Critical Hi All, Could any one please help me in this, I am getting below issue when my application is running in production environment. Production environment --------------------------------- I have two machine and each machine have 2 nodes. 1) Machine-1 have two nodes i.e. corenode1 and corenode2 2) Machine-2 have two nodes corenode3 and corenode4. Issue Reported ----------------------- Application is running smoothly but i have observed that sometime application will be very slow (Slow response issue has raised and when i checked JBoss application server log i could see below error). We have only option to restart server and after that application will be fine. I will appreciate if some one can assist me in this. Thanks Manish Error1: Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id ttb8-vdc0tdai-1-gkuzs3ai-p0mxki-al481a java.lang.IllegalStateException: Cannot find object in dispatcher with id ttb8-vdc0tdai-1-gkuzs3ai-p0mxki-al481a ====================================================================================================== org.jboss.jms.exception.MessagingJMSException: Failed to invoke 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$start$aop(ClientConnectionDelegate.java:264) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeTarget(ClientConnectionDelegate$start_N8025343665958530775.java) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.ConnectionAspect.handleStart(ConnectionAspect.java:147) 2015-06-01 08:52:08,785 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor388.invoke(Unknown Source) 2015-06-01 08:52:08,785 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 2015-06-01 08:52:08,785 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:703) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:758) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:244) Error2 ======================= ERROR [org.jboss.messaging.core.impl.postoffice.UnbindRequest] Failed to remove binding java.lang.IllegalStateException: Don't know about node id: 3 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBindingFromCluster(MessagingPostOffice.java:1912) at org.jboss.messaging.core.impl.postoffice.UnbindRequest.execute(UnbindRequest.java:61) at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlRequestHandler.handle(GroupMember.java:725) at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:616) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775) at org.jgroups.JChannel.up(JChannel.java:1336) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454) at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:494) at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153) at org.jgroups.protocols.FRAG2.up(FRAG2.java:188) at org.jgroups.protocols.FC.up(FC.java:473) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821) at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233) at org.jgroups.protocols.UNICAST.up(UNICAST.java:328) at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708) at org.jgroups.protocols.BARRIER.up(BARRIER.java:136) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:176) at org.jgroups.protocols.FD.up(FD.java:284) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328) at org.jgroups.protocols.MERGE2.up(MERGE2.java:144) at org.jgroups.protocols.Discovery.up(Discovery.java:264) at org.jgroups.protocols.PING.up(PING.java:273) at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319) at org.jgroups.protocols.TP.passMessageUp(TP.java:1253) at org.jgroups.protocols.TP.access$100(TP.java:50) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2015-06-01 04:27:02,017 ERROR [org.jboss.messaging.core.impl.postoffice.GroupMember] Caught Exception in RequestHandler java.lang.Exception: Failed to remove binding at org.jboss.messaging.core.impl.postoffice.UnbindRequest.execute(UnbindRequest.java:67) at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlRequestHandler.handle(GroupMember.java:725) at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:616) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775) at org.jgroups.JChannel.up(JChannel.java:1336) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454) at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:494) at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153) at org.jgroups.protocols.FRAG2.up(FRAG2.java:188) at org.jgroups.protocols.FC.up(FC.java:473) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821) at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233) at org.jgroups.protocols.UNICAST.up(UNICAST.java:328) at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708) at org.jgroups.protocols.BARRIER.up(BARRIER.java:136) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:176) at org.jgroups.protocols.FD.up(FD.java:284) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328) at org.jgroups.protocols.MERGE2.up(MERGE2.java:144) at org.jgroups.protocols.Discovery.up(Discovery.java:264) at org.jgroups.protocols.PING.up(PING.java:273) at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319) at org.jgroups.protocols.TP.passMessageUp(TP.java:1253) at org.jgroups.protocols.TP.access$100(TP.java:50) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalStateException: Don't know about node id: 3 Error3: ================== org.jboss.jms.exception.MessagingJMSException: Failed to invoke ============================================================== 2015-06-01 08:51:52,940 INFO [STDOUT] ********* E N D O F E R R O R R E P O R T ********* 2015-06-01 08:51:54,035 ERROR [STDERR] org.jboss.jms.exception.MessagingJMSException: Failed to invoke 2015-06-01 08:51:54,036 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271) 2015-06-01 08:51:54,036 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205) 2015-06-01 08:51:54,036 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$start$aop(ClientConnectionDelegate.java:264) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeTarget(ClientConnectionDelegate$start_N8025343665958530775.java) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.container.ConnectionAspect.handleStart(ConnectionAspect.java:147) 2015-06-01 08:51:54,037 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor388.invoke(Unknown Source) 2015-06-01 08:51:54,037 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 2015-06-01 08:51:54,037 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:703) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:758) 2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:244) Error4: ================== Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id ttb8-vdc0tdai-1-gkuzs3ai-p0mxki-al481a java.lang.IllegalStateException: Cannot find object in dispatcher with id ttb8-vdc0tdai-1-gkuzs3ai-p0mxki-al481a ====================================================================================================== org.jboss.jms.exception.MessagingJMSException: Failed to invoke 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$start$aop(ClientConnectionDelegate.java:264) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeTarget(ClientConnectionDelegate$start_N8025343665958530775.java) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.ConnectionAspect.handleStart(ConnectionAspect.java:147) 2015-06-01 08:52:08,785 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor388.invoke(Unknown Source) 2015-06-01 08:52:08,785 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 2015-06-01 08:52:08,785 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:703) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:758) 2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:244)