Failures in cluster cause significat disruption (IllegalStateException, "Rolling back to cache view Y, but last committed view is Z", TOs) ------------------------------------------------------------------------------------------------------------------------------------------ Key: AS7-4144 URL: https://issues.jboss.org/browse/AS7-4144 Project: Application Server 7 Issue Type: Bug Components: Clustering Affects Versions: 7.1.1.Final Environment: Linux Reporter: Radoslav Husar Assignee: Paul Ferraro Priority: Critical Labels: ISPN000071, ISPN000109, ISPN000152, ISPN000164, ISPN000170, eap6_need_triage, failover_testing Fix For: 7.1.2.Final-redhat1 Negative failover scenarios (server crashing) cause significant distruptions in the cluster. {noformat} [JBossINF] 17:19:13,423 INFO [org.jboss.weld.deployer] (MSC service thread 1-6) JBAS016002: Processing weld deployment clusterbench-ee6-web.war [JBossINF] 17:19:13,433 INFO [org.jboss.weld.deployer] (MSC service thread 1-15) JBAS016005: Starting Services for CDI deployment: clusterbench-ee6.ear [JBossINF] 17:19:13,527 INFO [org.jboss.weld.Version] (MSC service thread 1-15) WELD-000900 2012-02-22 16:46 [JBossINF] 17:19:13,552 INFO [org.jboss.weld.deployer] (MSC service thread 1-12) JBAS016008: Starting weld service for deployment clusterbench-ee6.ear [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) GMS: address=perf18/web, cluster=web, physical address=10.16.90.54:55200 [JBossINF] 17:19:13,866 INFO [stdout] (pool-13-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) GMS: address=perf18/ejb, cluster=ejb, physical address=10.16.90.54:55200 [JBossINF] 17:19:13,870 INFO [stdout] (pool-19-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:14,195 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-12) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,195 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-11) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,268 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000078: Starting JGroups Channel [JBossINF] 17:19:14,268 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000078: Starting JGroups Channel [JBossINF] 17:19:14,276 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000094: Received new cluster view: [perf19/ejb|5] [perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb] [JBossINF] 17:19:14,276 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000094: Received new cluster view: [perf19/web|5] [perf19/web, perf21/web, perf20/web, perf18/web] [JBossINF] 17:19:14,277 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000079: Cache local address is perf18/web, physical addresses are [10.16.90.54:55200] [JBossINF] 17:19:14,277 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000079: Cache local address is perf18/ejb, physical addresses are [10.16.90.54:55200] [JBossINF] 17:19:14,283 INFO [org.infinispan.factories.GlobalComponentRegistry] (pool-21-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL [JBossINF] 17:19:14,283 INFO [org.infinispan.factories.GlobalComponentRegistry] (pool-15-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL [JBossINF] 17:19:14,284 INFO [org.infinispan.config.ConfigurationValidatingVisitor] (pool-15-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,422 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-21-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,442 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-20-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,447 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-15-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-21-thread-1) JBAS010281: Started remote-connector-client-mappings cache from ejb container [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-15-thread-1) JBAS010281: Started repl cache from web container [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-20-thread-1) JBAS010281: Started repl cache from ejb container [JBossINF] 17:19:14,519 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-2) JBAS010206: Number of cluster members: 4 [JBossINF] 17:19:14,519 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (MSC service thread 1-12) JBAS010206: Number of cluster members: 4 [JBossINF] 17:19:14,555 INFO [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-3-thread-1) Added a new EJB receiver in cluster context ejb for node perf18. Total nodes in cluster context = 1 [JBossINF] 17:19:14,573 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,576 INFO [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,584 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-5) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,589 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-9) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,592 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-12) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,605 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-12) JBAS010281: Started org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl cache from ejb container [JBossINF] 17:19:14,606 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010281: Started org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container 2012/03/09 17:20:09:956 EST [DEBUG][Thread-33] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! [JBossINF] 17:20:13,087 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "clusterbench-ee6.ear" was rolled back with failure message Operation cancelled [JBossINF] 17:20:13,087 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment. [JBossINF] 17:20:13,095 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000029: Passivating all entries to disk [JBossINF] 17:20:13,096 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-16) ISPN000029: Passivating all entries to disk [JBossINF] 17:20:13,098 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000030: Passivated 0 entries in 3 milliseconds [JBossINF] 17:20:13,098 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-16) ISPN000030: Passivated 0 entries in 2 milliseconds [JBossINF] 17:20:13,106 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container [JBossINF] 17:20:13,106 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-16) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl cache from ejb container [JBossINF] 17:20:14,587 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010281: Started //default-host//clusterbench cache from web container [JBossINF] 17:20:14,596 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:20:14,597 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:20:14,844 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-18,null) JBAS010248: New cluster view for partition web: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@1a6fa30c delta: -1, merge: false) [JBossINF] 17:20:14,844 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (Incoming-11,null) JBAS010248: New cluster view for partition ejb: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@17397663 delta: -1, merge: false) [JBossINF] 17:20:14,846 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-18,null) ISPN000094: Received new cluster view: [perf21/web|6] [perf21/web, perf20/web, perf18/web] [JBossINF] 17:20:14,849 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-11,null) ISPN000094: Received new cluster view: [perf21/ejb|6] [perf21/ejb, perf20/ejb, perf18/ejb] [JBossINF] 17:20:14,974 INFO [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-19,null) ISPN000170: Our last committed view (CacheView{viewId=6, members=[perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=7, members=[perf19/ejb, perf21/ejb, perf20/ejb]}). This is normal during a merge [JBossINF] 17:20:14,977 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-15,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=//default-host//clusterbench, type=PREPARE_VIEW, sender=perf21/web, newViewId=8, newMembers=[perf21/web, perf20/web, perf18/web], oldViewId=7, oldMembers=[perf19/web, perf21/web, perf20/web]}: java.lang.IllegalStateException: Cannot prepare new view CacheView{viewId=8, members=[perf21/web, perf20/web, perf18/web]} on cache //default-host//clusterbench, we are currently preparing view CacheView{viewId=6, members=[perf19/web, perf21/web, perf20/web, perf18/web]} [JBossINF] at org.infinispan.cacheviews.CacheViewInfo.prepareView(CacheViewInfo.java:102) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:482) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:14,975 INFO [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-17,null) ISPN000170: Our last committed view (CacheView{viewId=6, members=[perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=7, members=[perf19/ejb, perf21/ejb, perf20/ejb]}). This is normal during a merge [JBossINF] 17:20:14,990 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-19,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl, type=PREPARE_VIEW, sender=perf21/ejb, newViewId=11, newMembers=[perf20/ejb, perf18/ejb, perf21/ejb], oldViewId=7, oldMembers=[perf19/ejb, perf21/ejb, perf20/ejb]}: java.lang.IllegalStateException: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl: Received cache view prepare request after the local node has already shut down [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:15,021 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-17,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB, type=PREPARE_VIEW, sender=perf21/ejb, newViewId=11, newMembers=[perf20/ejb, perf18/ejb, perf21/ejb], oldViewId=7, oldMembers=[perf19/ejb, perf21/ejb, perf20/ejb]}: java.lang.IllegalStateException: org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB: Received cache view prepare request after the local node has already shut down [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:15,072 ERROR [org.infinispan.cacheviews.CacheViewInfo] (OOB-17,null) ISPN000164: Rolling back to cache view 7, but last committed view is 6 [JBossINF] 17:20:15,084 ERROR [org.infinispan.cacheviews.CacheViewInfo] (OOB-17,null) ISPN000164: Rolling back to cache view 7, but last committed view is 6 [JBossINF] 17:20:15,197 INFO [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-5) Initializing Mojarra 2.1.5 (SNAPSHOT 20120217) for context '/clusterbench' [JBossINF] 17:20:16,638 INFO [org.hibernate.validator.util.Version] (MSC service thread 1-5) Hibernate Validator 4.2.0.Final-redhat-1 [JBossINF] 17:20:16,856 INFO [org.jboss.web] (MSC service thread 1-5) JBAS018210: Registering web context: /clusterbench [JBossINF] 17:21:15,196 WARN [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-14,null) ISPN000167: Rejecting state pushed by node perf21/web for view 8, there is no state transfer in progress (we are at view 9) [JBossINF] 17:21:15,196 WARN [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-20,null) ISPN000167: Rejecting state pushed by node perf20/web for view 8, there is no state transfer in progress (we are at view 9) [JBossINF] 17:21:16,864 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-13) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 10 [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:21:16,889 ERROR [org.infinispan.transaction.TransactionCoordinator] (MSC service thread 1-13) Error while processing prepare: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 10 [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [jbossweb-7.0.10.Final-redhat-1.jar:] [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:21:16,906 ERROR [org.infinispan.transaction.tm.DummyTransaction] (MSC service thread 1-13) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=6}, status=1}, lockedKeys=null, backupKeyLocks=null, viewId=6} org.infinispan.transaction.synchronization.SyncLocalTransaction@5} org.infinispan.transaction.synchronization.SynchronizationAdapter@24: org.infinispan.CacheException: Could not prepare. [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [jbossweb-7.0.10.Final-redhat-1.jar:] [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] Caused by: javax.transaction.xa.XAException [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:160) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [JBossINF] ... 27 more [JBossINF] {noformat}

Failures in cluster cause significat disruption (IllegalStateException, "Rolling back to cache view Y, but last committed view is Z", TOs) ------------------------------------------------------------------------------------------------------------------------------------------ Key: AS7-4144 URL: https://issues.jboss.org/browse/AS7-4144 Project: Application Server 7 Issue Type: Bug Components: Clustering Affects Versions: 7.1.1.Final Environment: Linux Reporter: Radoslav Husar Assignee: Paul Ferraro Priority: Critical Labels: ISPN000071, ISPN000109, ISPN000152, ISPN000164, ISPN000170, eap6_need_triage, failover_testing Fix For: 7.1.2.Final-redhat1 Negative failover scenarios (server crashing) cause significant distruptions in the cluster. {noformat} [JBossINF] 17:19:13,423 INFO [org.jboss.weld.deployer] (MSC service thread 1-6) JBAS016002: Processing weld deployment clusterbench-ee6-web.war [JBossINF] 17:19:13,433 INFO [org.jboss.weld.deployer] (MSC service thread 1-15) JBAS016005: Starting Services for CDI deployment: clusterbench-ee6.ear [JBossINF] 17:19:13,527 INFO [org.jboss.weld.Version] (MSC service thread 1-15) WELD-000900 2012-02-22 16:46 [JBossINF] 17:19:13,552 INFO [org.jboss.weld.deployer] (MSC service thread 1-12) JBAS016008: Starting weld service for deployment clusterbench-ee6.ear [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) GMS: address=perf18/web, cluster=web, physical address=10.16.90.54:55200 [JBossINF] 17:19:13,866 INFO [stdout] (pool-13-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) GMS: address=perf18/ejb, cluster=ejb, physical address=10.16.90.54:55200 [JBossINF] 17:19:13,870 INFO [stdout] (pool-19-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:14,195 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-12) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,195 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-11) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,268 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000078: Starting JGroups Channel [JBossINF] 17:19:14,268 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000078: Starting JGroups Channel [JBossINF] 17:19:14,276 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000094: Received new cluster view: [perf19/ejb|5] [perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb] [JBossINF] 17:19:14,276 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000094: Received new cluster view: [perf19/web|5] [perf19/web, perf21/web, perf20/web, perf18/web] [JBossINF] 17:19:14,277 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000079: Cache local address is perf18/web, physical addresses are [10.16.90.54:55200] [JBossINF] 17:19:14,277 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000079: Cache local address is perf18/ejb, physical addresses are [10.16.90.54:55200] [JBossINF] 17:19:14,283 INFO [org.infinispan.factories.GlobalComponentRegistry] (pool-21-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL [JBossINF] 17:19:14,283 INFO [org.infinispan.factories.GlobalComponentRegistry] (pool-15-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL [JBossINF] 17:19:14,284 INFO [org.infinispan.config.ConfigurationValidatingVisitor] (pool-15-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,422 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-21-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,442 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-20-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,447 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-15-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-21-thread-1) JBAS010281: Started remote-connector-client-mappings cache from ejb container [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-15-thread-1) JBAS010281: Started repl cache from web container [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-20-thread-1) JBAS010281: Started repl cache from ejb container [JBossINF] 17:19:14,519 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-2) JBAS010206: Number of cluster members: 4 [JBossINF] 17:19:14,519 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (MSC service thread 1-12) JBAS010206: Number of cluster members: 4 [JBossINF] 17:19:14,555 INFO [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-3-thread-1) Added a new EJB receiver in cluster context ejb for node perf18. Total nodes in cluster context = 1 [JBossINF] 17:19:14,573 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,576 INFO [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,584 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-5) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,589 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-9) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,592 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-12) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,605 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-12) JBAS010281: Started org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl cache from ejb container [JBossINF] 17:19:14,606 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010281: Started org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container 2012/03/09 17:20:09:956 EST [DEBUG][Thread-33] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! [JBossINF] 17:20:13,087 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "clusterbench-ee6.ear" was rolled back with failure message Operation cancelled [JBossINF] 17:20:13,087 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment. [JBossINF] 17:20:13,095 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000029: Passivating all entries to disk [JBossINF] 17:20:13,096 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-16) ISPN000029: Passivating all entries to disk [JBossINF] 17:20:13,098 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000030: Passivated 0 entries in 3 milliseconds [JBossINF] 17:20:13,098 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-16) ISPN000030: Passivated 0 entries in 2 milliseconds [JBossINF] 17:20:13,106 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container [JBossINF] 17:20:13,106 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-16) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl cache from ejb container [JBossINF] 17:20:14,587 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010281: Started //default-host//clusterbench cache from web container [JBossINF] 17:20:14,596 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:20:14,597 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:20:14,844 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-18,null) JBAS010248: New cluster view for partition web: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@1a6fa30c delta: -1, merge: false) [JBossINF] 17:20:14,844 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (Incoming-11,null) JBAS010248: New cluster view for partition ejb: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@17397663 delta: -1, merge: false) [JBossINF] 17:20:14,846 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-18,null) ISPN000094: Received new cluster view: [perf21/web|6] [perf21/web, perf20/web, perf18/web] [JBossINF] 17:20:14,849 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-11,null) ISPN000094: Received new cluster view: [perf21/ejb|6] [perf21/ejb, perf20/ejb, perf18/ejb] [JBossINF] 17:20:14,974 INFO [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-19,null) ISPN000170: Our last committed view (CacheView{viewId=6, members=[perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=7, members=[perf19/ejb, perf21/ejb, perf20/ejb]}). This is normal during a merge [JBossINF] 17:20:14,977 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-15,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=//default-host//clusterbench, type=PREPARE_VIEW, sender=perf21/web, newViewId=8, newMembers=[perf21/web, perf20/web, perf18/web], oldViewId=7, oldMembers=[perf19/web, perf21/web, perf20/web]}: java.lang.IllegalStateException: Cannot prepare new view CacheView{viewId=8, members=[perf21/web, perf20/web, perf18/web]} on cache //default-host//clusterbench, we are currently preparing view CacheView{viewId=6, members=[perf19/web, perf21/web, perf20/web, perf18/web]} [JBossINF] at org.infinispan.cacheviews.CacheViewInfo.prepareView(CacheViewInfo.java:102) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:482) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:14,975 INFO [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-17,null) ISPN000170: Our last committed view (CacheView{viewId=6, members=[perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=7, members=[perf19/ejb, perf21/ejb, perf20/ejb]}). This is normal during a merge [JBossINF] 17:20:14,990 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-19,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl, type=PREPARE_VIEW, sender=perf21/ejb, newViewId=11, newMembers=[perf20/ejb, perf18/ejb, perf21/ejb], oldViewId=7, oldMembers=[perf19/ejb, perf21/ejb, perf20/ejb]}: java.lang.IllegalStateException: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl: Received cache view prepare request after the local node has already shut down [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:15,021 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-17,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB, type=PREPARE_VIEW, sender=perf21/ejb, newViewId=11, newMembers=[perf20/ejb, perf18/ejb, perf21/ejb], oldViewId=7, oldMembers=[perf19/ejb, perf21/ejb, perf20/ejb]}: java.lang.IllegalStateException: org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB: Received cache view prepare request after the local node has already shut down [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:15,072 ERROR [org.infinispan.cacheviews.CacheViewInfo] (OOB-17,null) ISPN000164: Rolling back to cache view 7, but last committed view is 6 [JBossINF] 17:20:15,084 ERROR [org.infinispan.cacheviews.CacheViewInfo] (OOB-17,null) ISPN000164: Rolling back to cache view 7, but last committed view is 6 [JBossINF] 17:20:15,197 INFO [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-5) Initializing Mojarra 2.1.5 (SNAPSHOT 20120217) for context '/clusterbench' [JBossINF] 17:20:16,638 INFO [org.hibernate.validator.util.Version] (MSC service thread 1-5) Hibernate Validator 4.2.0.Final-redhat-1 [JBossINF] 17:20:16,856 INFO [org.jboss.web] (MSC service thread 1-5) JBAS018210: Registering web context: /clusterbench [JBossINF] 17:21:15,196 WARN [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-14,null) ISPN000167: Rejecting state pushed by node perf21/web for view 8, there is no state transfer in progress (we are at view 9) [JBossINF] 17:21:15,196 WARN [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-20,null) ISPN000167: Rejecting state pushed by node perf20/web for view 8, there is no state transfer in progress (we are at view 9) [JBossINF] 17:21:16,864 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-13) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 10 [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:21:16,889 ERROR [org.infinispan.transaction.TransactionCoordinator] (MSC service thread 1-13) Error while processing prepare: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 10 [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [jbossweb-7.0.10.Final-redhat-1.jar:] [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:21:16,906 ERROR [org.infinispan.transaction.tm.DummyTransaction] (MSC service thread 1-13) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=6}, status=1}, lockedKeys=null, backupKeyLocks=null, viewId=6} org.infinispan.transaction.synchronization.SyncLocalTransaction@5} org.infinispan.transaction.synchronization.SynchronizationAdapter@24: org.infinispan.CacheException: Could not prepare. [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [jbossweb-7.0.10.Final-redhat-1.jar:] [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] Caused by: javax.transaction.xa.XAException [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:160) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [JBossINF] ... 27 more [JBossINF] {noformat}