[jboss-jira] [JBoss JIRA] (AS7-4144) Failures in cluster cause significat disruption (IllegalStateException, "Rolling back to cache view Y, but last committed view is Z", TOs)

Failures in cluster cause significat disruption (IllegalStateException, "Rolling back to cache view Y, but last committed view is Z", TOs) ------------------------------------------------------------------------------------------------------------------------------------------ Key: AS7-4144 URL: https://issues.jboss.org/browse/AS7-4144 Project: Application Server 7 Issue Type: Bug Components: Clustering Affects Versions: 7.1.1.Final Environment: Linux Reporter: Radoslav Husar Assignee: Paul Ferraro Priority: Critical Labels: ISPN000071, ISPN000109, ISPN000152, ISPN000164, ISPN000170, eap6_need_triage, failover_testing Fix For: 7.1.2.Final-redhat1 Negative failover scenarios (server crashing) cause significant distruptions in the cluster. {noformat} [JBossINF] 17:19:13,423 INFO [org.jboss.weld.deployer] (MSC service thread 1-6) JBAS016002: Processing weld deployment clusterbench-ee6-web.war [JBossINF] 17:19:13,433 INFO [org.jboss.weld.deployer] (MSC service thread 1-15) JBAS016005: Starting Services for CDI deployment: clusterbench-ee6.ear [JBossINF] 17:19:13,527 INFO [org.jboss.weld.Version] (MSC service thread 1-15) WELD-000900 2012-02-22 16:46 [JBossINF] 17:19:13,552 INFO [org.jboss.weld.deployer] (MSC service thread 1-12) JBAS016008: Starting weld service for deployment clusterbench-ee6.ear [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,865 INFO [stdout] (pool-13-thread-1) GMS: address=perf18/web, cluster=web, physical address=10.16.90.54:55200 [JBossINF] 17:19:13,866 INFO [stdout] (pool-13-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:13,869 INFO [stdout] (pool-19-thread-1) GMS: address=perf18/ejb, cluster=ejb, physical address=10.16.90.54:55200 [JBossINF] 17:19:13,870 INFO [stdout] (pool-19-thread-1) ------------------------------------------------------------------- [JBossINF] 17:19:14,195 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-12) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,195 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-11) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,268 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000078: Starting JGroups Channel [JBossINF] 17:19:14,268 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000078: Starting JGroups Channel [JBossINF] 17:19:14,276 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000094: Received new cluster view: [perf19/ejb|5] [perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb] [JBossINF] 17:19:14,276 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000094: Received new cluster view: [perf19/web|5] [perf19/web, perf21/web, perf20/web, perf18/web] [JBossINF] 17:19:14,277 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000079: Cache local address is perf18/web, physical addresses are [10.16.90.54:55200] [JBossINF] 17:19:14,277 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000079: Cache local address is perf18/ejb, physical addresses are [10.16.90.54:55200] [JBossINF] 17:19:14,283 INFO [org.infinispan.factories.GlobalComponentRegistry] (pool-21-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL [JBossINF] 17:19:14,283 INFO [org.infinispan.factories.GlobalComponentRegistry] (pool-15-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL [JBossINF] 17:19:14,284 INFO [org.infinispan.config.ConfigurationValidatingVisitor] (pool-15-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,422 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-21-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,442 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-20-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,447 INFO [org.infinispan.jmx.CacheJmxRegistration] (pool-15-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-21-thread-1) JBAS010281: Started remote-connector-client-mappings cache from ejb container [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-15-thread-1) JBAS010281: Started repl cache from web container [JBossINF] 17:19:14,503 INFO [org.jboss.as.clustering.infinispan] (pool-20-thread-1) JBAS010281: Started repl cache from ejb container [JBossINF] 17:19:14,519 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-2) JBAS010206: Number of cluster members: 4 [JBossINF] 17:19:14,519 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (MSC service thread 1-12) JBAS010206: Number of cluster members: 4 [JBossINF] 17:19:14,555 INFO [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-3-thread-1) Added a new EJB receiver in cluster context ejb for node perf18. Total nodes in cluster context = 1 [JBossINF] 17:19:14,573 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,576 INFO [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:19:14,584 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-5) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,589 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-9) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,592 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-12) ISPN000031: MBeans were successfully registered to the platform mbean server. [JBossINF] 17:19:14,605 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-12) JBAS010281: Started org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl cache from ejb container [JBossINF] 17:19:14,606 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010281: Started org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container 2012/03/09 17:20:09:956 EST [DEBUG][Thread-33] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! [JBossINF] 17:20:13,087 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "clusterbench-ee6.ear" was rolled back with failure message Operation cancelled [JBossINF] 17:20:13,087 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment. [JBossINF] 17:20:13,095 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000029: Passivating all entries to disk [JBossINF] 17:20:13,096 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-16) ISPN000029: Passivating all entries to disk [JBossINF] 17:20:13,098 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000030: Passivated 0 entries in 3 milliseconds [JBossINF] 17:20:13,098 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-16) ISPN000030: Passivated 0 entries in 2 milliseconds [JBossINF] 17:20:13,106 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container [JBossINF] 17:20:13,106 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-16) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl cache from ejb container [JBossINF] 17:20:14,587 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010281: Started //default-host//clusterbench cache from web container [JBossINF] 17:20:14,596 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:20:14,597 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated. [JBossINF] 17:20:14,844 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-18,null) JBAS010248: New cluster view for partition web: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@1a6fa30c delta: -1, merge: false) [JBossINF] 17:20:14,844 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (Incoming-11,null) JBAS010248: New cluster view for partition ejb: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@17397663 delta: -1, merge: false) [JBossINF] 17:20:14,846 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-18,null) ISPN000094: Received new cluster view: [perf21/web|6] [perf21/web, perf20/web, perf18/web] [JBossINF] 17:20:14,849 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-11,null) ISPN000094: Received new cluster view: [perf21/ejb|6] [perf21/ejb, perf20/ejb, perf18/ejb] [JBossINF] 17:20:14,974 INFO [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-19,null) ISPN000170: Our last committed view (CacheView{viewId=6, members=[perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=7, members=[perf19/ejb, perf21/ejb, perf20/ejb]}). This is normal during a merge [JBossINF] 17:20:14,977 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-15,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=//default-host//clusterbench, type=PREPARE_VIEW, sender=perf21/web, newViewId=8, newMembers=[perf21/web, perf20/web, perf18/web], oldViewId=7, oldMembers=[perf19/web, perf21/web, perf20/web]}: java.lang.IllegalStateException: Cannot prepare new view CacheView{viewId=8, members=[perf21/web, perf20/web, perf18/web]} on cache //default-host//clusterbench, we are currently preparing view CacheView{viewId=6, members=[perf19/web, perf21/web, perf20/web, perf18/web]} [JBossINF] at org.infinispan.cacheviews.CacheViewInfo.prepareView(CacheViewInfo.java:102) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:482) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:14,975 INFO [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-17,null) ISPN000170: Our last committed view (CacheView{viewId=6, members=[perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=7, members=[perf19/ejb, perf21/ejb, perf20/ejb]}). This is normal during a merge [JBossINF] 17:20:14,990 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-19,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl, type=PREPARE_VIEW, sender=perf21/ejb, newViewId=11, newMembers=[perf20/ejb, perf18/ejb, perf21/ejb], oldViewId=7, oldMembers=[perf19/ejb, perf21/ejb, perf20/ejb]}: java.lang.IllegalStateException: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl: Received cache view prepare request after the local node has already shut down [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:15,021 WARN [org.infinispan.commands.control.CacheViewControlCommand] (OOB-17,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB, type=PREPARE_VIEW, sender=perf21/ejb, newViewId=11, newMembers=[perf20/ejb, perf18/ejb, perf21/ejb], oldViewId=7, oldMembers=[perf19/ejb, perf21/ejb, perf20/ejb]}: java.lang.IllegalStateException: org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB: Received cache view prepare request after the local node has already shut down [JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:20:15,072 ERROR [org.infinispan.cacheviews.CacheViewInfo] (OOB-17,null) ISPN000164: Rolling back to cache view 7, but last committed view is 6 [JBossINF] 17:20:15,084 ERROR [org.infinispan.cacheviews.CacheViewInfo] (OOB-17,null) ISPN000164: Rolling back to cache view 7, but last committed view is 6 [JBossINF] 17:20:15,197 INFO [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-5) Initializing Mojarra 2.1.5 (SNAPSHOT 20120217) for context '/clusterbench' [JBossINF] 17:20:16,638 INFO [org.hibernate.validator.util.Version] (MSC service thread 1-5) Hibernate Validator 4.2.0.Final-redhat-1 [JBossINF] 17:20:16,856 INFO [org.jboss.web] (MSC service thread 1-5) JBAS018210: Registering web context: /clusterbench [JBossINF] 17:21:15,196 WARN [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-14,null) ISPN000167: Rejecting state pushed by node perf21/web for view 8, there is no state transfer in progress (we are at view 9) [JBossINF] 17:21:15,196 WARN [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-20,null) ISPN000167: Rejecting state pushed by node perf20/web for view 8, there is no state transfer in progress (we are at view 9) [JBossINF] 17:21:16,864 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-13) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 10 [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:21:16,889 ERROR [org.infinispan.transaction.TransactionCoordinator] (MSC service thread 1-13) Error while processing prepare: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 10 [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [jbossweb-7.0.10.Final-redhat-1.jar:] [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] [JBossINF] 17:21:16,906 ERROR [org.infinispan.transaction.tm.DummyTransaction] (MSC service thread 1-13) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=6}, status=1}, lockedKeys=null, backupKeyLocks=null, viewId=6} org.infinispan.transaction.synchronization.SyncLocalTransaction@5} org.infinispan.transaction.synchronization.SynchronizationAdapter@24: org.infinispan.CacheException: Could not prepare. [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [JBossINF] at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [JBossINF] at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [JBossINF] at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [jbossweb-7.0.10.Final-redhat-1.jar:] [JBossINF] at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] Caused by: javax.transaction.xa.XAException [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:160) [JBossINF] at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [JBossINF] ... 27 more [JBossINF] {noformat}