[infinispan-dev] state transfer exceptions at REPL

Dan Berindei dan.berindei at gmail.com
Tue Feb 7 07:54:23 EST 2012


Sanne, this sounds very similar to
https://issues.jboss.org/browse/ISPN-1814, but I thought I had fixed
that for 5.1.1.FINAL.

I see CacheViewsManagerImpl is trying to install a view with 6 nodes,
should there be 6 nodes in the cluster or should there be less nodes?
Do you have DEBUG logs for org.infinispan and org.jgroups?

Cheers
Dan


On Tue, Feb 7, 2012 at 12:58 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
> Can anyone explain this error?
>
> I'm updating Hibernate Search, and having a simple test which in a loop does:
>
> - write to shared index
> - add a node / remove a node
> - wait for joins
> - verifies index state
>
> This is expected to work, as it already did with all previous
> Infinispan versions.
>
> Using Infinispan 5.1.1.FINAL and JGroups 3.0.5.Final.
>
> 2012-02-07 10:42:38,668 WARN  [CacheViewControlCommand]
> (OOB-4,sanne-20017) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesMetadata,
> type=PREPARE_VIEW, sender=sanne-3158, newViewId=8,
> newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158,
> sanne-63971, sanne-20017, sanne-2794, sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
>        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
>        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
>        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
>        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
>        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
>        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
>        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
>        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
>        at org.jgroups.JChannel.up(JChannel.java:716)
>        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
>        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
>        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
>        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
>        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
>        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
>        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
>        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
>        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
>        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
>        at org.jgroups.protocols.Discovery.up(Discovery.java:355)
>        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
>        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
>        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
>        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
>        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        ... 3 more
> 2012-02-07 10:42:38,706 WARN  [CacheViewControlCommand]
> (OOB-5,sanne-20017) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW,
> sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971,
> sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7,
> oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
>        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
>        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
>        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
>        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
>        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
>        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
>        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
>        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
>        at org.jgroups.JChannel.up(JChannel.java:716)
>        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
>        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
>        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
>        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
>        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
>        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
>        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
>        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
>        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
>        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
>        at org.jgroups.protocols.Discovery.up(Discovery.java:355)
>        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
>        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
>        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
>        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
>        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        ... 3 more
> 2012-02-07 10:42:38,684 WARN  [UNICAST2] (OOB-7,sanne-2794)
> sanne-2794: my conn_id (6) != received conn_id (1); discarding STABLE
> message !
> 2012-02-07 10:42:38,671 WARN  [CacheViewControlCommand]
> (OOB-3,sanne-63971) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesMetadata,
> type=PREPARE_VIEW, sender=sanne-3158, newViewId=8,
> newMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511, sanne-30075], oldViewId=7, oldMembers=[sanne-3158,
> sanne-63971, sanne-20017, sanne-2794, sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-63971, viewId=8, state=24}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
>        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
>        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
>        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
>        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
>        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
>        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
>        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
>        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
>        at org.jgroups.JChannel.up(JChannel.java:716)
>        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
>        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
>        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
>        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
>        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
>        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
>        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
>        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
>        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
>        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
>        at org.jgroups.protocols.Discovery.up(Discovery.java:355)
>        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
>        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
>        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-63971, viewId=8, state=24}
>        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
>        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        ... 3 more
> 2012-02-07 10:42:38,677 WARN  [CacheViewControlCommand]
> (OOB-4,sanne-63971) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW,
> sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971,
> sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7,
> oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-63971, viewId=8, state=22}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
>        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
>        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
>        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
>        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
>        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
>        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
>        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
>        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
>        at org.jgroups.JChannel.up(JChannel.java:716)
>        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
>        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
>        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
>        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
>        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
>        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
>        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
>        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
>        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
>        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
>        at org.jgroups.protocols.Discovery.up(Discovery.java:355)
>        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
>        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
>        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-63971, viewId=8, state=22}
>        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
>        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        ... 3 more
> 2012-02-07 10:42:38,718 WARN  [CacheViewControlCommand]
> (OOB-6,sanne-25511) ISPN000071: Caught exception when handling command
> CacheViewControlCommand{cache=LuceneIndexesData, type=PREPARE_VIEW,
> sender=sanne-3158, newViewId=8, newMembers=[sanne-3158, sanne-63971,
> sanne-20017, sanne-2794, sanne-25511, sanne-30075], oldViewId=7,
> oldMembers=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511]}
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-25511, viewId=8, state=19}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
>        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
>        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
>        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
>        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
>        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
>        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
>        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
>        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
>        at org.jgroups.JChannel.up(JChannel.java:716)
>        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
>        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
>        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
>        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
>        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
>        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
>        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
>        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
>        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
>        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
>        at org.jgroups.protocols.Discovery.up(Discovery.java:355)
>        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
>        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
>        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-25511, viewId=8, state=19}
>        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
>        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        ... 3 more
> 2012-02-07 10:42:38,733 ERROR [CacheViewsManagerImpl]
> (CacheViewInstaller-1,sanne-3158) ISPN000172: Failed to prepare view
> CacheView{viewId=8, members=[sanne-3158, sanne-63971, sanne-20017,
> sanne-2794, sanne-25511, sanne-30075]} for cache
> LuceneIndexesMetadata, rolling back to view CacheView{viewId=7,
> members=[sanne-3158, sanne-63971, sanne-20017, sanne-2794,
> sanne-25511]}
> java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:319)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
>        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
>        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
>        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
>        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
>        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
>        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
>        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
>        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
>        at org.jgroups.JChannel.up(JChannel.java:716)
>        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
>        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
>        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
>        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
>        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
>        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
>        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
>        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
>        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
>        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
>        at org.jgroups.protocols.Discovery.up(Discovery.java:355)
>        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
>        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
>        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
>        ... 3 more
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesMetadata,
> type=APPLY_STATE, sender=sanne-20017, viewId=8, state=4}
>        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
>        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        ... 3 more
> 2012-02-07 10:42:38,737 ERROR [CacheViewsManagerImpl]
> (CacheViewInstaller-3,sanne-3158) ISPN000172: Failed to prepare view
> CacheView{viewId=8, members=[sanne-3158, sanne-63971, sanne-20017,
> sanne-2794, sanne-25511, sanne-30075]} for cache  LuceneIndexesData,
> rolling back to view CacheView{viewId=7, members=[sanne-3158,
> sanne-63971, sanne-20017, sanne-2794, sanne-25511]}
> java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:319)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException:
> org.infinispan.remoting.transport.jgroups.SuspectException: One or
> more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>        at org.infinispan.util.concurrent.AggregatingNotifyingFutureBuilder.get(AggregatingNotifyingFutureBuilder.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferTask.finishPushingState(BaseStateTransferTask.java:139)
>        at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:116)
>        at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93)
>        at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:294)
>        at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:486)
>        at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125)
>        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161)
>        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141)
>        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
>        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
>        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
>        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
>        at org.jgroups.JChannel.up(JChannel.java:716)
>        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
>        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
>        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
>        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
>        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
>        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
>        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
>        at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
>        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)
>        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
>        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
>        at org.jgroups.protocols.Discovery.up(Discovery.java:355)
>        at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
>        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
>        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)
>        ... 3 more
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException:
> One or more nodes have left the cluster while replicating command
> StateTransferControlCommand{cache=LuceneIndexesData, type=APPLY_STATE,
> sender=sanne-20017, viewId=8, state=3}
>        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)
>        at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:78)
>        at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:249)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        ... 3 more
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



More information about the infinispan-dev mailing list