Re: [jboss-as7-dev] Infinispan subsystem error if clustered application is deployed in AS7.2

Message: 1 Date: Thu, 15 Nov 2012 18:29:59 +0100 From: Wolf-Dieter Fink <wfink(a)redhat.com&gt; Subject: [jboss-as7-dev] Infinispan subsystem error if clustered application is deployed in AS7.2 To: JBoss AS7 Development <jboss-as7-dev(a)lists.jboss.org&gt; Message-ID: <50A52697.1040700(a)redhat.com&gt; Content-Type: text/plain; charset=ISO-8859-1; format=flowed I've seen some strange behaviour in AS7.2 (upstream). If there is a clustered application deployed I see an Exception [1] during shutdown. In this case a shutdown will be longer than normal [2] Also a re-deployment of this clustered application might fail and rolled-back. The cluster has two nodes app-oneA and app-oneB. What can be the reason for that? - Wolf [1] [Host Controller] 17:57:39,788 INFO [org.jboss.as.repository] (management-handler-thread - 5) JBAS014900: Content added at location /data/srv/eap/6.0.0/test/domain/data/content/23/879179e47349cb024c78c2f02dcfbf908d6d51/content [Server:app-oneA] 17:57:39,814 INFO [org.jboss.as.osgi] (MSC service thread 1-15) JBAS011908: Unregister module: Module "deployment.appone.ear:main" from Service Module Loader [Server:app-oneA] 17:57:39,815 INFO [org.jboss.as.osgi] (MSC service thread 1-11) JBAS011908: Unregister module: Module "deployment.appone.ear.ejb.jar:main" from Service Module Loader [Server:app-oneB] 17:57:39,818 INFO [org.jboss.as.osgi] (MSC service thread 1-3) JBAS011908: Unregister module: Module "deployment.appone.ear:main" from Service Module Loader [Server:app-oneB] 17:57:39,819 INFO [org.jboss.as.osgi] (MSC service thread 1-12) JBAS011908: Unregister module: Module "deployment.appone.ear.ejb.jar:main" from Service Module Loader [Server:app-oneA] 17:57:39,829 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-11) ISPN000029: Passivating all entries to disk [Server:app-oneA] 17:57:39,829 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-11) ISPN000030: Passivated 0 entries in 0 milliseconds [Server:app-oneB] 17:57:39,832 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-14) ISPN000029: Passivating all entries to disk [Server:app-oneB] 17:57:39,834 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-14) ISPN000030: Passivated 0 entries in 1 milliseconds [Server:app-oneB] 17:57:39,838 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015877: Stopped deployment ejb.jar in 26ms [Server:app-oneA] 17:57:39,840 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-11) JBAS010282: Stopped repl cache from ejb container [Server:app-oneA] 17:57:39,840 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-16) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container [Server:app-oneB] 17:57:39,840 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015877: Stopped deployment appone.ear in 28ms [Server:app-oneB] 17:57:39,840 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-14) JBAS010282: Stopped repl cache from ejb container [Server:app-oneA] 17:57:39,841 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment ejb.jar in 28ms [Server:app-oneB] 17:57:39,841 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-19,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=repl, type=PREPARE_VIEW, sender=master:app-oneA/ejb, newViewId=3, newMembers=[master:app-oneB/ejb], oldViewId=2, oldMembers=[master:app-oneA/ejb, master:app-oneB/ejb]}: java.lang.IllegalStateException: repl: Received cache view prepare request after the local node has already shut down [Server:app-oneB] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:487) [Server:app-oneB] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:126) [Server:app-oneB] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:221) [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:201) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.RSVP.up(RSVP.java:179) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.stack.Protocol.up(Protocol.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] [Server:app-oneB] 17:57:39,844 INFO [org.jboss.as.server.deployment] (MSC service thread 1-12) JBAS015876: Starting deployment of "appone.ear" [Server:app-oneA] 17:57:39,848 INFO [org.jboss.as.server.deployment] (MSC service thread 1-9) JBAS015877: Stopped deployment appone.ear in 36ms [Server:app-oneB] 17:57:39,849 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) JBAS015876: Starting deployment of "ejb.jar" [Server:app-oneA] 17:57:39,850 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "appone.ear" [Server:app-oneA] 17:57:39,854 INFO [org.jboss.as.server.deployment] (MSC service thread 1-14) JBAS015876: Starting deployment of "ejb.jar" [Server:app-oneA] 17:57:39,864 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-3,master:app-oneA/ejb) ISPN000172: Failed to prepare view CacheView{viewId=3, members=[master:app-oneB/ejb]} for cache repl, rolling back to view CacheView{viewId=2, members=[master:app-oneA/ejb, master:app-oneB/ejb]}: java.util.concurrent.ExecutionException: org.infinispan.CacheException: java.lang.IllegalStateException: repl: Received cache view prepare request after the local node has already shut down [Server:app-oneA] at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.util.concurrent.FutureTask.get(FutureTask.java:119) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:318) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:249) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:875) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] Caused by: org.infinispan.CacheException: java.lang.IllegalStateException: repl: Received cache view prepare request after the local node has already shut down [Server:app-oneA] at org.infinispan.util.Util.rewrapAsCacheException(Util.java:525) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:172) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:489) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.cacheviews.CacheViewsManagerImpl$2.call(CacheViewsManagerImpl.java:302) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.cacheviews.CacheViewsManagerImpl$2.call(CacheViewsManagerImpl.java:299) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] ... 5 more [Server:app-oneA] Caused by: java.lang.IllegalStateException: repl: Received cache view prepare request after the local node has already shut down [Server:app-oneA] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:487) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:126) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:221) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:201) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.RSVP.up(RSVP.java:179) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.stack.Protocol.up(Protocol.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] ... 3 more [Server:app-oneA] [Server:app-oneB] 17:57:39,873 INFO [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-11) JNDI bindings for session bean named AppOneBean in deployment unit subdeployment "ejb.jar" of deployment "appone.ear" are as follows: [Server:app-oneB] [Server:app-oneB] java:global/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne [Server:app-oneB] java:app/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne [Server:app-oneB] java:module/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne [Server:app-oneB] java:jboss/exported/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne [Server:app-oneB] java:global/appone/ejb/AppOneBean [Server:app-oneB] java:app/ejb/AppOneBean [Server:app-oneB] java:module/AppOneBean [Server:app-oneB] [Server:app-oneB] 17:57:39,877 INFO [org.jboss.as.osgi] (MSC service thread 1-12) JBAS011907: Register module: Module "deployment.appone.ear:main" from Service Module Loader [Server:app-oneA] 17:57:39,881 INFO [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-14) JNDI bindings for session bean named AppOneBean in deployment unit subdeployment "ejb.jar" of deployment "appone.ear" are as follows: [Server:app-oneA] [Server:app-oneA] java:global/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne [Server:app-oneA] java:app/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne [Server:app-oneA] java:module/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne [Server:app-oneA] java:jboss/exported/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne [Server:app-oneA] java:global/appone/ejb/AppOneBean [Server:app-oneA] java:app/ejb/AppOneBean [Server:app-oneA] java:module/AppOneBean [Server:app-oneA] [Server:app-oneA] 17:57:39,886 INFO [org.jboss.as.osgi] (MSC service thread 1-12) JBAS011907: Register module: Module "deployment.appone.ear:main" from Service Module Loader [Server:app-oneA] 17:57:39,902 INFO [org.jboss.as.osgi] (MSC service thread 1-7) JBAS011907: Register module: Module "deployment.appone.ear.ejb.jar:main" from Service Module Loader [Server:app-oneB] 17:57:39,912 INFO [org.jboss.as.osgi] (MSC service thread 1-5) JBAS011907: Register module: Module "deployment.appone.ear.ejb.jar:main" from Service Module Loader [Server:app-oneA] 17:57:39,925 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000082: Stopping the RpcDispatcher [Server:app-oneB] 17:57:39,930 INFO [org.infinispan.jmx.CacheJmxRegistration] (CacheService lifecycle - 2) ISPN000031: MBeans were successfully registered to the platform mbean server. [Server:app-oneA] 17:57:39,932 WARN [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-18,null) Problems unmarshalling remote command from byte buffer: java.lang.NullPointerException [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154) [Server:app-oneA] at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) [Server:app-oneA] at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85) [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.RSVP.up(RSVP.java:179) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.stack.Protocol.up(Protocol.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] [Server:app-oneB] 17:57:39,956 ERROR [org.jboss.msc.service.fail] (CacheService lifecycle - 2) MSC000001: Failed to start service jboss.infinispan.ejb.repl: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.repl: org.infinispan.CacheException: Unable to invoke method private void org.infinispan.statetransfer.BaseStateTransferManagerImpl.start() throws java.lang.Exception on object of type ReplicatedStateTransferManagerImpl [Server:app-oneB] at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:68) [jboss-as-clustering-common-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-1.jar:2.0.0.GA-redhat-1] [Server:app-oneB] Caused by: org.infinispan.CacheException: Unable to invoke method private void org.infinispan.statetransfer.BaseStateTransferManagerImpl.start() throws java.lang.Exception on object of type ReplicatedStateTransferManagerImpl [Server:app-oneB] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:885) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:628) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:531) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:174) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.CacheImpl.start(CacheImpl.java:521) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.AbstractDelegatingCache.start(AbstractDelegatingCache.java:343) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:77) [jboss-as-clustering-infinispan-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:65) [jboss-as-clustering-common-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] ... 4 more [Server:app-oneB] Caused by: org.infinispan.CacheException: java.util.concurrent.ExecutionException: org.infinispan.marshall.NotSerializableException: org.infinispan.remoting.responses.ExceptionResponse [Server:app-oneB] at org.infinispan.util.Util.rewrapAsCacheException(Util.java:525) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:139) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:492) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.cacheviews.CacheViewsManagerImpl.join(CacheViewsManagerImpl.java:213) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.statetransfer.BaseStateTransferManagerImpl.start(BaseStateTransferManagerImpl.java:146) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:236) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] ... 13 more [Server:app-oneB] Caused by: java.util.concurrent.ExecutionException: org.infinispan.marshall.NotSerializableException: org.infinispan.remoting.responses.ExceptionResponse [Server:app-oneB] at org.jgroups.blocks.UnicastRequest.getValue(UnicastRequest.java:165) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.jgroups.blocks.UnicastRequest.get(UnicastRequest.java:191) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:331) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:132) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] ... 21 more [Server:app-oneB] Caused by: org.infinispan.marshall.NotSerializableException: org.infinispan.remoting.responses.ExceptionResponse [Server:app-oneB] Caused by: an exception which occurred: [Server:app-oneB] in object org.infinispan.remoting.responses.ExceptionResponse@40a5e467 [Server:app-oneB] [Server:app-oneA] 17:57:39,979 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (CacheService lifecycle - 2) ISPN000078: Starting JGroups Channel [Server:app-oneA] 17:57:39,980 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (CacheService lifecycle - 2) ISPN000094: Received new cluster view: [master:app-oneA/ejb|1] [master:app-oneA/ejb, master:app-oneB/ejb] [Server:app-oneA] 17:57:39,980 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (CacheService lifecycle - 2) ISPN000079: Cache local address is master:app-oneA/ejb, physical addresses are [127.0.0.1:55300] [Server:app-oneA] 17:57:39,988 INFO [org.infinispan.jmx.CacheJmxRegistration] (CacheService lifecycle - 2) ISPN000031: MBeans were successfully registered to the platform mbean server. [Server:app-oneA] 17:57:39,991 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 2) JBAS010281: Started remote-connector-client-mappings cache from ejb container [Server:app-oneA] 17:57:40,001 INFO [org.infinispan.jmx.CacheJmxRegistration] (CacheService lifecycle - 2) ISPN000031: MBeans were successfully registered to the platform mbean server. [Server:app-oneA] 17:57:40,002 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 2) JBAS010281: Started repl cache from ejb container [Server:app-oneA] 17:57:40,003 INFO [org.jboss.as.clustering] (MSC service thread 1-9) JBAS010225: New cluster view for partition ejb (id: 1, delta: 0, merge: false) : [master:app-oneA/ejb, master:app-oneB/ejb] [Server:app-oneB] 17:58:09,839 WARN [org.infinispan.transaction.TransactionTable] (MSC service thread 1-16) ISPN000100: Stopping, but there are 0 local transactions and 1 remote transactions that did not finish in time. [Server:app-oneB] 17:58:09,842 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-16) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container [Server:app-oneB] 17:58:09,853 INFO [org.infinispan.jmx.CacheJmxRegistration] (CacheService lifecycle - 2) ISPN000031: MBeans were successfully registered to the platform mbean server. [Server:app-oneB] 17:58:09,857 INFO [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-19,null) ISPN000170: Our last committed view (CacheView{viewId=3, members=[master:app-oneB/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=2, members=[master:app-oneA/ejb]}). This is normal during a merge [Server:app-oneB] 17:58:09,859 WARN [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-18,null) ISPN000167: Rejecting state pushed by node master:app-oneA/ejb for view 3, there is no state transfer in progress (we are at view 3) [Server:app-oneB] 17:58:09,862 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 2) JBAS010281: Started remote-connector-client-mappings cache from ejb container [Server:app-oneA] 17:58:09,866 WARN [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-18,null) Problems unmarshalling remote command from byte buffer: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0 [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:893) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [Server:app-oneA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107) [Server:app-oneA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58) [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [Server:app-oneA] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:163) [Server:app-oneA] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:67) [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154) [Server:app-oneA] at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) [Server:app-oneA] at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85) [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.RSVP.up(RSVP.java:192) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.stack.Protocol.up(Protocol.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] [Server:app-oneB] 17:58:09,892 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (RegistryService lifecycle - 2) ISPN000136: Execution error: org.infinispan.CacheException: Problems invoking command. [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:210) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [Server:app-oneB] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [Server:app-oneB] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [Server:app-oneB] at org.jgroups.JChannel.up(JChannel.java:716) [Server:app-oneB] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [Server:app-oneB] at org.jgroups.protocols.RSVP.up(RSVP.java:192) [Server:app-oneB] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [Server:app-oneB] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [Server:app-oneB] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [Server:app-oneB] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [Server:app-oneB] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [Server:app-oneB] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [Server:app-oneB] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [Server:app-oneB] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [Server:app-oneB] at org.jgroups.protocols.FD.up(FD.java:273) [Server:app-oneB] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [Server:app-oneB] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [Server:app-oneB] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [Server:app-oneB] at org.jgroups.stack.Protocol.up(Protocol.java:363) [Server:app-oneB] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] Caused by: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0 [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:893) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:163) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:67) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] ... 29 more [Server:app-oneB] [Server:app-oneB] 17:58:09,901 ERROR [org.infinispan.transaction.TransactionCoordinator] (RegistryService lifecycle - 2) Error while processing prepare: org.infinispan.CacheException: Problems invoking command. [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:210) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [Server:app-oneB] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [Server:app-oneB] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [Server:app-oneB] at org.jgroups.JChannel.up(JChannel.java:716) [Server:app-oneB] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [Server:app-oneB] at org.jgroups.protocols.RSVP.up(RSVP.java:192) [Server:app-oneB] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [Server:app-oneB] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [Server:app-oneB] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [Server:app-oneB] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [Server:app-oneB] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [Server:app-oneB] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [Server:app-oneB] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [Server:app-oneB] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [Server:app-oneB] at org.jgroups.protocols.FD.up(FD.java:273) [Server:app-oneB] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [Server:app-oneB] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [Server:app-oneB] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [Server:app-oneB] at org.jgroups.stack.Protocol.up(Protocol.java:363) [Server:app-oneB] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] Caused by: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0 [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:893) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:163) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:67) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] ... 29 more [Server:app-oneB] [Server:app-oneB] 17:58:09,905 ERROR [org.infinispan.transaction.tm.DummyTransaction] (RegistryService lifecycle - 2) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=3}, status=1}, lockedKeys=null, backupKeyLocks=[master:app-oneB/ejb], viewId=1} org.infinispan.transaction.synchronization.SyncLocalTransaction@3} org.infinispan.transaction.synchronization.SynchronizationAdapter@22: org.infinispan.CacheException: Could not prepare. [Server:app-oneB] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:241) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:258) [jboss-as-clustering-registry-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.registry.RegistryService.refreshLocalEntry(RegistryService.java:130) [jboss-as-clustering-registry-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.registry.RegistryService.start(RegistryService.java:113) [jboss-as-clustering-registry-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:65) [jboss-as-clustering-common-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [Server:app-oneB] Caused by: javax.transaction.xa.XAException [Server:app-oneB] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:160) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneA] 17:58:09,908 WARN [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-18,null) Problems unmarshalling remote command from byte buffer: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0 [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:893) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [Server:app-oneA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107) [Server:app-oneA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58) [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [Server:app-oneA] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:163) [Server:app-oneA] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:67) [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [Server:app-oneA] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA-redhat-1.jar:1.3.14.GA-redhat-1] [Server:app-oneA] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154) [Server:app-oneA] at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) [Server:app-oneA] at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85) [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [Server:app-oneA] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.RSVP.up(RSVP.java:192) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.stack.Protocol.up(Protocol.java:363) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1] [Server:app-oneA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneA] [Server:app-oneB] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] ... 17 more [Server:app-oneB] [Server:app-oneB] 17:58:09,917 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (RegistryService lifecycle - 2) ISPN000136: Execution error: org.infinispan.CacheException: Problems invoking command. [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:210) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [Server:app-oneB] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [Server:app-oneB] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [Server:app-oneB] at org.jgroups.JChannel.up(JChannel.java:716) [Server:app-oneB] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [Server:app-oneB] at org.jgroups.protocols.RSVP.up(RSVP.java:192) [Server:app-oneB] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [Server:app-oneB] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [Server:app-oneB] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [Server:app-oneB] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [Server:app-oneB] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [Server:app-oneB] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [Server:app-oneB] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [Server:app-oneB] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [Server:app-oneB] at org.jgroups.protocols.FD.up(FD.java:273) [Server:app-oneB] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [Server:app-oneB] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [Server:app-oneB] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [Server:app-oneB] at org.jgroups.stack.Protocol.up(Protocol.java:363) [Server:app-oneB] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] Caused by: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0 [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:893) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:163) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:67) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] ... 29 more [Server:app-oneB] [Server:app-oneB] 17:58:09,922 ERROR [org.infinispan.transaction.TransactionCoordinator] (RegistryService lifecycle - 2) ISPN000188: Error while processing a commit in a two-phase transaction: org.infinispan.CacheException: Problems invoking command. [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:210) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [Server:app-oneB] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [Server:app-oneB] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [Server:app-oneB] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [Server:app-oneB] at org.jgroups.JChannel.up(JChannel.java:716) [Server:app-oneB] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [Server:app-oneB] at org.jgroups.protocols.RSVP.up(RSVP.java:192) [Server:app-oneB] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [Server:app-oneB] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [Server:app-oneB] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [Server:app-oneB] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [Server:app-oneB] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [Server:app-oneB] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [Server:app-oneB] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [Server:app-oneB] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [Server:app-oneB] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [Server:app-oneB] at org.jgroups.protocols.FD.up(FD.java:273) [Server:app-oneB] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [Server:app-oneB] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [Server:app-oneB] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [Server:app-oneB] at org.jgroups.stack.Protocol.up(Protocol.java:363) [Server:app-oneB] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [Server:app-oneB] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] Caused by: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0 [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:893) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:163) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:67) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [Server:app-oneB] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [Server:app-oneB] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [Server:app-oneB] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] ... 29 more [Server:app-oneB] [Server:app-oneB] 17:58:09,929 ERROR [org.infinispan.transaction.tm.DummyTransaction] (RegistryService lifecycle - 2) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=3}, status=3}, lockedKeys=null, backupKeyLocks=[master:app-oneB/ejb], viewId=1} org.infinispan.transaction.synchronization.SyncLocalTransaction@3} org.infinispan.transaction.synchronization.SynchronizationAdapter@22: org.infinispan.CacheException: Could not commit. [Server:app-oneB] at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:279) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:328) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:258) [jboss-as-clustering-registry-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.registry.RegistryService.refreshLocalEntry(RegistryService.java:130) [jboss-as-clustering-registry-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.registry.RegistryService.start(RegistryService.java:113) [jboss-as-clustering-registry-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:65) [jboss-as-clustering-common-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] [Server:app-oneB] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [Server:app-oneB] Caused by: javax.transaction.xa.XAException [Server:app-oneB] at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:223) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:184) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.1.4.FINAL-redhat-1.jar:5.1.4.FINAL-redhat-1] [Server:app-oneB] ... 17 more [Server:app-oneB] [2] [Server:app-main] 18:18:30,184 INFO [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-10) HornetQ Server version 2.2.16.Final (HQ_2_2_16_FINAL, 122) [e4024d6b-2f3e-11e2-89ee-e747e1e36f1c] stopped [Server:app-main] 18:18:30,185 INFO [org.jboss.as] (MSC service thread 1-14) JBAS015950: JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1) stopped in 224ms 18:18:30,191 INFO [org.jboss.as.process.Server:app-main.status] (reaper for Server:app-main) JBAS012010: Process 'Server:app-main' finished with an exit status of 130 [Server:app-oneB] 18:18:30,367 INFO [org.jboss.as] (MSC service thread 1-15) JBAS015950: JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1) stopped in 493ms 18:18:30,372 INFO [org.jboss.as.process.Server:app-oneB.status] (reaper for Server:app-oneB) JBAS012010: Process 'Server:app-oneB' finished with an exit status of 130 [Server:app-oneA] 18:18:45,038 INFO [org.jboss.as] (MSC service thread 1-1) JBAS015950: JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1) stopped in 15088ms 18:18:45,051 INFO [org.jboss.as.process.Server:app-oneA.status] (reaper for Server:app-oneA) JBAS012010: Process 'Server:app-oneA' finished with an exit status of 130 [Host Controller] 18:18:45,135 INFO [org.jboss.as] (MSC service thread 1-2) JBAS015950: JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1) stopped in 15265ms 18:18:45,139 INFO [org.jboss.as.process.Host Controller.status] (reaper for Host Controller) JBAS012010: Process 'Host Controller' finished with an exit status of 130 18:18:45,140 INFO [org.jboss.as.process] (Shutdown thread) JBAS012015: All processes finished; exiting ------------------------------ _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev End of jboss-as7-dev Digest, Vol 25, Issue 15 *********************************************