Can anyone explain this error? I'm updating Hibernate Search, and having a simple test which in a loop does: - write to shared index - add a node / remove a node - wait for joins - verifies index state This is expected to work, as it already did with all previous Infinispan versions. Using Infinispan 5.1.1.FINAL and JGroups 3.0.5.Final. 2012-02-07 10:42:38,668 WARN  [CacheViewControlCommand] (OOB-4,sanne-20017) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=LuceneIndexesMetadata, type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511]} java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesMetadata, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)        at org.jgroups.JChannel.up(JChannel.java:716)        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)        at org.jgroups.protocols.Discovery.up(Discovery.java:355)        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesMetadata, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        ... 3 more 2012-02-07 10:42:38,706 WARN  [CacheViewControlCommand] (OOB-5,sanne-20017) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511]} java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=3}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)        at org.jgroups.JChannel.up(JChannel.java:716)        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)        at org.jgroups.protocols.Discovery.up(Discovery.java:355)        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=3}        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        ... 3 more 2012-02-07 10:42:38,684 WARN  [UNICAST2] (OOB-7,sanne-2794) sanne-2794: my conn_id (6) != received conn_id (1); discarding STABLE message ! 2012-02-07 10:42:38,671 WARN  [CacheViewControlCommand] (OOB-3,sanne-63971) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=LuceneIndexesMetadata, type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511]} java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesMetadata, type=APPLY_STATE, sender=sanne-63971, viewId=8, state=24}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)        at org.jgroups.JChannel.up(JChannel.java:716)        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)        at org.jgroups.protocols.Discovery.up(Discovery.java:355)        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesMetadata, type=APPLY_STATE, sender=sanne-63971, viewId=8, state=24}        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        ... 3 more 2012-02-07 10:42:38,677 WARN  [CacheViewControlCommand] (OOB-4,sanne-63971) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511]} java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-63971, viewId=8, state=22}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)        at org.jgroups.JChannel.up(JChannel.java:716)        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)        at org.jgroups.protocols.Discovery.up(Discovery.java:355)        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-63971, viewId=8, state=22}        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        ... 3 more 2012-02-07 10:42:38,718 WARN  [CacheViewControlCommand] (OOB-6,sanne-25511) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511]} java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-25511, viewId=8, state=19}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)        at org.jgroups.JChannel.up(JChannel.java:716)        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)        at org.jgroups.protocols.Discovery.up(Discovery.java:355)        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-25511, viewId=8, state=19}        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        ... 3 more 2012-02-07 10:42:38,733 ERROR [CacheViewsManagerImpl] (CacheViewInstaller-1,sanne-3158) ISPN000172: Failed to prepare view CacheView{viewId=8, members=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511, sanne-30075]} for cache LuceneIndexesMetadata, rolling back to view CacheView{viewId=7, members=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511]} java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesMetadata, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:319)        at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)        at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesMetadata, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)        at org.jgroups.JChannel.up(JChannel.java:716)        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)        at org.jgroups.protocols.Discovery.up(Discovery.java:355)        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)        ... 3 more Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesMetadata, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        ... 3 more 2012-02-07 10:42:38,737 ERROR [CacheViewsManagerImpl] (CacheViewInstaller-3,sanne-3158) ISPN000172: Failed to prepare view CacheView{viewId=8, members=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511, sanne-30075]} for cache  LuceneIndexesData, rolling back to view CacheView{viewId=7, members=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, sanne-25511]} java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=3}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:319)        at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)        at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=3}        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)        at java.util.concurrent.FutureTask.get(FutureTask.java:91)        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)        at org.jgroups.JChannel.up(JChannel.java:716)        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)        at org.jgroups.protocols.Discovery.up(Discovery.java:355)        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)        ... 3 more Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE, sender=sanne-20017, viewId=8, state=3}        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        ... 3 more _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I am also seeing this/similar exception in AS7 during session replication even with 5.1.1.FINAL :-( On 02/07/2012 01:54 PM, Dan Berindei wrote: > Sanne, this sounds very similar to > https://issues.jboss.org/browse/ISPN-1814, but I thought I had fixed > that for 5.1.1.FINAL. > > I see CacheViewsManagerImpl is trying to install a view with 6 nodes, > should there be 6 nodes in the cluster or should there be less nodes? > Do you have DEBUG logs for org.infinispan and org.jgroups? > > Cheers > Dan > > > On Tue, Feb 7, 2012 at 12:58 PM, Sanne Grinovero<sanne(a)infinispan.org>  wrote: >> Can anyone explain this error? >> >> I'm updating Hibernate Search, and having a simple test which in a loop does: >> >> - write to shared index >> - add a node / remove a node >> - wait for joins >> - verifies index state >> >> This is expected to work, as it already did with all previous >> Infinispan versions. >> >> Using Infinispan 5.1.1.FINAL and JGroups 3.0.5.Final. >> >> 2012-02-07 10:42:38,668 WARN  [CacheViewControlCommand] >> (OOB-4,sanne-20017) ISPN000071: Caught exception when handling command >> CacheViewControlCommand{cache=LuceneIndexesMetadata, >> type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, >> newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, >> sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, >> sanne-63971, sanne-20017, sanne-2794, sanne-25511]} >> java.util.concurrent.ExecutionException: >> org.infinispan.remoting.transport.jgroups.SuspectException: One or >> more nodes have left the cluster while replicating command >> StateTransferControlCommand{cache=LuceneIndexesMetadata, >> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4} _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I am also seeing this/similar exception in AS7 during session replication even with 5.1.1.FINAL :-( On 02/07/2012 01:54 PM, Dan Berindei wrote: > Sanne, this sounds very similar to > https://issues.jboss.org/browse/ISPN-1814, but I thought I had fixed > that for 5.1.1.FINAL. > > I see CacheViewsManagerImpl is trying to install a view with 6 nodes, > should there be 6 nodes in the cluster or should there be less nodes? > Do you have DEBUG logs for org.infinispan and org.jgroups? > > Cheers > Dan > > > On Tue, Feb 7, 2012 at 12:58 PM, Sanne Grinovero<sanne(a)infinispan.org>  wrote: >> Can anyone explain this error? >> >> I'm updating Hibernate Search, and having a simple test which in a loop does: >> >> - write to shared index >> - add a node / remove a node >> - wait for joins >> - verifies index state >> >> This is expected to work, as it already did with all previous >> Infinispan versions. >> >> Using Infinispan 5.1.1.FINAL and JGroups 3.0.5.Final. >> >> 2012-02-07 10:42:38,668 WARN  [CacheViewControlCommand] >> (OOB-4,sanne-20017) ISPN000071: Caught exception when handling command >> CacheViewControlCommand{cache=LuceneIndexesMetadata, >> type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, >> newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, >> sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, >> sanne-63971, sanne-20017, sanne-2794, sanne-25511]} >> java.util.concurrent.ExecutionException: >> org.infinispan.remoting.transport.jgroups.SuspectException: One or >> more nodes have left the cluster while replicating command >> StateTransferControlCommand{cache=LuceneIndexesMetadata, >> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4} _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Dan, you can easily checkout Hibernate Search, it's a Maven project and you should be able to set it up in your IDE quickly. git clone git://github.com/Sanne/hibernate-search.git git checkout componentsUpdates Then the failing test is in the module "hibernate-search-infinispan".. which is just a couple of classes. Sanne On 7 February 2012 16:10, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > Rado, is there a specific test in the AS7 test suite that is failing? > Is it only in Jenkins or on your machine as well? > > I only know about https://issues.jboss.org/browse/ISPN-1806, but Paul > said that he doesn't see it any more in CI runs (he never managed to > reproduce it on his machine). > > Cheers > Dan > > > On Tue, Feb 7, 2012 at 3:13 PM, Radoslav Husar <rhusar(a)redhat.com&gt; wrote: >> I am also seeing this/similar exception in AS7 during session >> replication even with 5.1.1.FINAL :-( >> >> On 02/07/2012 01:54 PM, Dan Berindei wrote: >>> Sanne, this sounds very similar to >>> https://issues.jboss.org/browse/ISPN-1814, but I thought I had fixed >>> that for 5.1.1.FINAL. >>> >>> I see CacheViewsManagerImpl is trying to install a view with 6 nodes, >>> should there be 6 nodes in the cluster or should there be less nodes? >>> Do you have DEBUG logs for org.infinispan and org.jgroups? >>> >>> Cheers >>> Dan >>> >>> >>> On Tue, Feb 7, 2012 at 12:58 PM, Sanne Grinovero&lt;sanne(a)infinispan.org&gt;  wrote: >>>> Can anyone explain this error? >>>> >>>> I'm updating Hibernate Search, and having a simple test which in a loop does: >>>> >>>> - write to shared index >>>> - add a node / remove a node >>>> - wait for joins >>>> - verifies index state >>>> >>>> This is expected to work, as it already did with all previous >>>> Infinispan versions. >>>> >>>> Using Infinispan 5.1.1.FINAL and JGroups 3.0.5.Final. >>>> >>>> 2012-02-07 10:42:38,668 WARN  [CacheViewControlCommand] >>>> (OOB-4,sanne-20017) ISPN000071: Caught exception when handling command >>>> CacheViewControlCommand{cache=LuceneIndexesMetadata, >>>> type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, >>>> newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, >>>> sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, >>>> sanne-63971, sanne-20017, sanne-2794, sanne-25511]} >>>> java.util.concurrent.ExecutionException: >>>> org.infinispan.remoting.transport.jgroups.SuspectException: One or >>>> more nodes have left the cluster while replicating command >>>> StateTransferControlCommand{cache=LuceneIndexesMetadata, >>>> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4} >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Sanne, I was able to run LiveRunningTest as well after I removed TestableJGroupsTransport from the Infinispan configuration, and I disabled queueing in the SHARED_LOOPBACK OOB thread pool:   <SHARED_LOOPBACK         thread_pool.enabled="true"         thread_pool.min_threads="2"         thread_pool.max_threads="30"         thread_pool.keep_alive_time="60000"         thread_pool.queue_enabled="false"         thread_pool.queue_max_size="100"         thread_pool.rejection_policy="Discard"         oob_thread_pool.enabled="true"         oob_thread_pool.min_threads="2"         oob_thread_pool.max_threads="30"         oob_thread_pool.keep_alive_time="60000"         oob_thread_pool.queue_enabled="false"         oob_thread_pool.queue_max_size="100"         oob_thread_pool.rejection_policy="Discard"         /> I think the test fails with queuing enabled and core thread pool size 2 because the coordinator sends a PREPARE_VIEW command and several APPLY_STATE commands (at least one for each cache) at approximately the same time. If two APPLY_STATE commands get to the other node before the PREPARE_VIEW command, they will be stuck waiting for state transfer to start. FD also sends messages using OOB, so if the OOB thread pool stops processing messages FD on other members will soon suspect the stuck member and kick it out of the cluster. For now I think increasing the number of available threads is the only solution. For 5.2 I'm thinking of moving both the sending of state and the handling of state to a separate thread, so that OOB threads won't have to block waiting for the state transfer to start. Cheers Dan On Wed, Feb 8, 2012 at 9:59 AM, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > Hi Sanne > > I got the sources and even TwoNodesTest hang for me every time. > > I think the problem is that your TestableJGroupsTransport is trying to > modify the cluster name during startup - which is no longer supported. > > I have also created https://issues.jboss.org/browse/ISPN-1852 to fix > startup so that after an error like this another getCache() call > doesn't block forever. Ideally it should report the same error, > whether we attempt to start the component again or we save the > exception somewhere. > > Cheers > Dan > > > On Tue, Feb 7, 2012 at 6:15 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: >> Dan, >> you can easily checkout Hibernate Search, it's a Maven project and you >> should be able to set it up in your IDE quickly. >> >> git clone git://github.com/Sanne/hibernate-search.git >> git checkout componentsUpdates >> >> Then the failing test is in the module "hibernate-search-infinispan".. >> which is just a couple of classes. >> >> Sanne >> >> >> >> On 7 February 2012 16:10, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >>> Rado, is there a specific test in the AS7 test suite that is failing? >>> Is it only in Jenkins or on your machine as well? >>> >>> I only know about https://issues.jboss.org/browse/ISPN-1806, but Paul >>> said that he doesn't see it any more in CI runs (he never managed to >>> reproduce it on his machine). >>> >>> Cheers >>> Dan >>> >>> >>> On Tue, Feb 7, 2012 at 3:13 PM, Radoslav Husar <rhusar(a)redhat.com&gt; wrote: >>>> I am also seeing this/similar exception in AS7 during session >>>> replication even with 5.1.1.FINAL :-( >>>> >>>> On 02/07/2012 01:54 PM, Dan Berindei wrote: >>>>> Sanne, this sounds very similar to >>>>> https://issues.jboss.org/browse/ISPN-1814, but I thought I had fixed >>>>> that for 5.1.1.FINAL. >>>>> >>>>> I see CacheViewsManagerImpl is trying to install a view with 6 nodes, >>>>> should there be 6 nodes in the cluster or should there be less nodes? >>>>> Do you have DEBUG logs for org.infinispan and org.jgroups? >>>>> >>>>> Cheers >>>>> Dan >>>>> >>>>> >>>>> On Tue, Feb 7, 2012 at 12:58 PM, Sanne Grinovero&lt;sanne(a)infinispan.org&gt;  wrote: >>>>>> Can anyone explain this error? >>>>>> >>>>>> I'm updating Hibernate Search, and having a simple test which in a loop does: >>>>>> >>>>>> - write to shared index >>>>>> - add a node / remove a node >>>>>> - wait for joins >>>>>> - verifies index state >>>>>> >>>>>> This is expected to work, as it already did with all previous >>>>>> Infinispan versions. >>>>>> >>>>>> Using Infinispan 5.1.1.FINAL and JGroups 3.0.5.Final. >>>>>> >>>>>> 2012-02-07 10:42:38,668 WARN  [CacheViewControlCommand] >>>>>> (OOB-4,sanne-20017) ISPN000071: Caught exception when handling command >>>>>> CacheViewControlCommand{cache=LuceneIndexesMetadata, >>>>>> type=PREPARE_VIEW, sender=sanne-3158, newViewId=8, >>>>>> newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794, >>>>>> sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158, >>>>>> sanne-63971, sanne-20017, sanne-2794, sanne-25511]} >>>>>> java.util.concurrent.ExecutionException: >>>>>> org.infinispan.remoting.transport.jgroups.SuspectException: One or >>>>>> more nodes have left the cluster while replicating command >>>>>> StateTransferControlCommand{cache=LuceneIndexesMetadata, >>>>>> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4} >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev