[infinispan-dev] state transfer exceptions at REPL
Dan Berindei
dan.berindei at gmail.com
Tue Feb 7 07:54:23 EST 2012
Sanne, this sounds very similar to
https://issues.jboss.org/browse/ISPN-1814, but I thought I had fixed
that for 5.1.1.FINAL.
I see CacheViewsManagerImpl is trying to install a view with 6 nodes,
should there be 6 nodes in the cluster or should there be less nodes?
Do you have DEBUG logs for org.infinispan and org.jgroups?
Cheers
Dan
On Tue, Feb 7, 2012 at 12:58 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
> Can anyone explain this error?
>
> I'm updating Hibernate Search, and having a simple test which in a loop does:
>
> - write to shared index
> - add a node / remove a node
> - wait for joins
> - verifies index state
>
> This is expected to work, as it already did with all previous
> Infinispan versions.
>
> Using Infinispan 5.1.1.FINAL and JGroups 3.0.5.Final.
>
> 2012-02-07 10:42:38,668 WARN [CacheViewControlCommand]
> (OOB-4,sanne-20017) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesMetadata,
> type=PREPARE_VIEW, sender=sanne-3158, newViewId=8,
> newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158,
> sanne-63971, sanne-20017, sanne-2794, sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
> at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
> at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
> at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
> at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
> at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
> at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:355)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> 2012-02-07 10:42:38,706 WARN [CacheViewControlCommand]
> (OOB-5,sanne-20017) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW,
> sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971,
> sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7,
> oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
> at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
> at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
> at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
> at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
> at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
> at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:355)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> 2012-02-07 10:42:38,684 WARN [UNICAST2] (OOB-7,sanne-2794)
> sanne-2794: my conn_id (6) != received conn_id (1); discarding STABLE
> message !
> 2012-02-07 10:42:38,671 WARN [CacheViewControlCommand]
> (OOB-3,sanne-63971) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesMetadata,
> type=PREPARE_VIEW, sender=sanne-3158, newViewId=8,
> newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158,
> sanne-63971, sanne-20017, sanne-2794, sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-63971, viewId=8, state=24}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
> at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
> at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
> at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
> at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
> at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
> at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:355)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-63971, viewId=8, state=24}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> 2012-02-07 10:42:38,677 WARN [CacheViewControlCommand]
> (OOB-4,sanne-63971) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW,
> sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971,
> sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7,
> oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-63971, viewId=8, state=22}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
> at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
> at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
> at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
> at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
> at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
> at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:355)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-63971, viewId=8, state=22}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> 2012-02-07 10:42:38,718 WARN [CacheViewControlCommand]
> (OOB-6,sanne-25511) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW,
> sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971,
> sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7,
> oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-25511, viewId=8, state=19}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
> at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
> at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
> at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
> at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
> at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
> at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:355)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-25511, viewId=8, state=19}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> 2012-02-07 10:42:38,733 ERROR [CacheViewsManagerImpl]
> (CacheViewInstaller-1,sanne-3158) ISPN000172: Failed to prepare view
> CacheView{viewId=8, members=[sanne-3158, sanne-63971, sanne-20017,
> sanne-2794, sanne-25511, sanne-30075]} for cache
> LuceneIndexesMetadata, rolling back to view CacheView{viewId=7,
> members=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511]}
> java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:319)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)
> at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
> at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
> at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
> at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
> at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
> at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
> at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:355)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
> ... 3 more
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> 2012-02-07 10:42:38,737 ERROR [CacheViewsManagerImpl]
> (CacheViewInstaller-3,sanne-3158) ISPN000172: Failed to prepare view
> CacheView{viewId=8, members=[sanne-3158, sanne-63971, sanne-20017,
> sanne-2794, sanne-25511, sanne-30075]} for cache LuceneIndexesData,
> rolling back to view CacheView{viewId=7, members=[sanne-3158,
> sanne-63971, sanne-20017, sanne-2794, sanne-25511]}
> java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:319)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)
> at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
> at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
> at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
> at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
> at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
> at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
> at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
> at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:355)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
> ... 3 more
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
More information about the infinispan-dev
mailing list