[JBoss JIRA] (ISPN-3120) StateConsumerImpl can ignore state received during a rebalance
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3120?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3120:
--------------------------------
Labels: 5.2.x nbst (was: 5.2.x)
> StateConsumerImpl can ignore state received during a rebalance
> --------------------------------------------------------------
>
> Key: ISPN-3120
> URL: https://issues.jboss.org/browse/ISPN-3120
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.3.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x, nbst
> Fix For: 5.2.7.Final, 5.3.0.CR1, 5.3.0.Final
>
>
> This causes random failures in ConcurrentOverlappingLeaveTest and ConcurrentNonOverlappingLeaveTest.
> 1. Starting with a 4-node cluster: [E, F, G, H] (topology 7).
> 2. F leaves, and E sends a REBALANCE_START command with nodes [E, G, H] (topology 8). Some segments are owned by [H] in the current CH and by [H, G] in the pending CH.
> 3. E reports that it finished receiving state with a REBAlANCE_CONFIRM command.
> 4. H leaves, and E sends a CH_UPDATE command with nodes [E, G] (topology 9).
> The segments that were owned by [H] in the previous currentCH are assigned to [E, G] in the new currentCH (otherwise they wouldn't have any owners).
> 5. The StateConsumerImpl on E requests state for the "lost" segments from G.
> 6. G confirms the end of the rebalance as well, and E sends a CH_UPDATE command to end the rebalance (topology 10).
> 7. E sends a REBALANCE_START command to assign all segments for [E, G] (topology 11).
> 8. While the StateConsumerImpl on E is starting the state transfer, it also receives a StateResponseCommand for the lost segments from G.
> 9. Because the structures keeping track of the received state are not properly initialized, E considers it finished receiving state for topology 11.
> 10. E receives a StateResponseCommand from G with actual data, but it ignores it because {{StateConsumerImpl.updatedKeys == null}}.
> {noformat}
> 11:30:39,807 DEBUG (transport-thread-4,NodeE:dist) [LocalTopologyManagerImpl] Updating local consistent hash(es) for cache dist: new topology = CacheTopology{id=7, currentCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeE-51027, NodeG-6339, NodeH-47370]}, pendingCH=null}
> 11:30:39,810 DEBUG (transport-thread-3,NodeE:dist) [LocalTopologyManagerImpl] Starting local rebalance for cache dist, topology = CacheTopology{id=8, currentCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeE-51027, NodeG-6339, NodeH-47370]}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeE-51027, NodeG-6339, NodeH-47370]}}
> 11:30:39,817 DEBUG (transport-thread-3,NodeE:dist) [StateConsumerImpl] Finished receiving of segments for cache dist for topology 8.
> 11:30:39,832 DEBUG (transport-thread-4,NodeE:dist) [LocalTopologyManagerImpl] Updating local consistent hash(es) for cache dist: new topology = CacheTopology{id=9, currentCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeE-51027, NodeG-6339]}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeE-51027, NodeG-6339]}}
> 11:30:39,834 DEBUG (transport-thread-4,NodeE:dist) [StateConsumerImpl] Adding inbound state transfer for segments [38, 36, 47, 44, 45] of cache dist
> 11:30:39,853 DEBUG (transport-thread-3,NodeE:dist) [LocalTopologyManagerImpl] Starting local rebalance for cache dist, topology = CacheTopology{id=11, currentCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeE-51027, NodeG-6339]}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeE-51027, NodeG-6339]}}
> 11:30:39,859 TRACE (remote-thread-1,NodeE:) [InboundInvocationHandlerImpl] Calling perform() on StateResponseCommand{cache=dist, origin=NodeG-6339, topologyId=9}
> 11:30:39,864 DEBUG (remote-thread-1,NodeE:dist) [StateConsumerImpl] Finished receiving of segments for cache dist for topology 11.
> 11:30:39,866 TRACE (transport-thread-5,NodeE:dist) [LocalTopologyManagerImpl] Ignoring consistent hash update 10 for cache dist, we have already received a newer topology 11
> 11:30:39,868 TRACE (remote-thread-1,NodeE:) [InboundInvocationHandlerImpl] Calling perform() on StateResponseCommand{cache=dist, origin=NodeG-6339, topologyId=11}
> 11:30:39,872 TRACE (remote-thread-1,NodeE:dist dist) [EntryWrappingInterceptor] State transfer will not write key/value MagicKey#k3{672f69c9@NodeG-6339}/v3 because it was already updated by somebody else
> 11:30:40,582 ERROR (testng-ConcurrentNonOverlappingLeaveTest:) [UnitTestTestNGListener] Test testTransactional(org.infinispan.distribution.rehash.ConcurrentNonOverlappingLeaveTest) failed.
> java.lang.AssertionError: Fail on owner cache NodeE-51027: dc.get(MagicKey#k3{672f69c9@NodeG-6339}) returned null!
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-3089) The first write to a joiner in invalidation mode can be ignored
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3089?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3089:
--------------------------------
Labels: nbst (was: )
> The first write to a joiner in invalidation mode can be ignored
> ---------------------------------------------------------------
>
> Key: ISPN-3089
> URL: https://issues.jboss.org/browse/ISPN-3089
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.6.Final, 5.3.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: nbst
> Fix For: 5.3.0.CR2, 5.3.0.Final
>
>
> In invalidation mode we don't wait for the initial state transfer to finish, or even for the joiner to become a member in the "write" CH, before returning to the user from {{getCache()}}.
> Writes to the joiner before it becomes a member of the write CH are not committed to the local cache, because of a check in {{EntryWrappingInterceptor.shouldWrap()}}. So it's possible for the joiner to ignore the first write completely.
> If {{StateTransferConfigurationBuilder}} enabled {{awaitInitialTransfer}} by default in invalidation mode, like it does in distributed/replicated mode, this wouldn't happen.
> The fact that it will still be possible for the user to disable {{awaitInitialTransfer}} manually shouldn't be a problem, because in invalidation mode the user should expect values to be invalidated at any time. This is just about improving the default configuration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-2781) NPE in AbstractTopologyAwareEncoder1x
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2781?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2781:
--------------------------------
Labels: nbst (was: )
> NPE in AbstractTopologyAwareEncoder1x
> -------------------------------------
>
> Key: ISPN-2781
> URL: https://issues.jboss.org/browse/ISPN-2781
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 5.2.0.CR3
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Labels: nbst
> Fix For: 5.2.2.Final, 5.3.0.Alpha1, 5.3.0.Final
>
>
> got this in an elasticity test (32nodes)
> {code}
> 06:23:20,660 ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-151) ISPN005009: Unexpected error before any request parameters read: java.lang.NullPointerException
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x$$anonfun$writeHashTopologyUpdate11$3.apply(AbstractTopologyAwareEncoder1x.scala:102) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x$$anonfun$writeHashTopologyUpdate11$3.apply(AbstractTopologyAwareEncoder1x.scala:101) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x.writeHashTopologyUpdate11(AbstractTopologyAwareEncoder1x.scala:101) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x.writeHashTopologyUpdate(AbstractTopologyAwareEncoder1x.scala:52) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.AbstractEncoder1x.writeHeader(AbstractEncoder1x.scala:63) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:63) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:66) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.Channels.write(Channels.java:704) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.Channels.write(Channels.java:671) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) [netty-3.6.1.Final.jar:]
> at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:179) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.HotRodDecoder.customDecodeHeader(HotRodDecoder.scala:157) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:105) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:70) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:47) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.1.Final.jar:]
> at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:387) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:313) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.1.Final.jar:]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-2897) NPE in DefaultConsistentHash
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2897?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2897:
--------------------------------
Labels: 5.2.x nbst (was: 5.2.x)
> NPE in DefaultConsistentHash
> ----------------------------
>
> Key: ISPN-2897
> URL: https://issues.jboss.org/browse/ISPN-2897
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.3.Final
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x, nbst
> Fix For: 5.2.4.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: serverlogs.zip
>
>
> Happened in local testing on my laptop with 4 nodes JDG 6.1.0.CR1 (ISPN 5.2.3.Final)
> For some reason there's a rebalance with pendingCH=null and this is not handled well by DefaultConsistentHash.union:
> {code}
> 15:11:04,782 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (OOB-5,shared=udp) Starting local rebalance for cache testCache, topology = CacheTopology{id=3, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[node04/default]}, pendingCH=null}
> 15:11:04,782 WARN [org.infinispan.topology.CacheTopologyControlCommand] (OOB-5,shared=udp) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=testCache, type=REBALANCE_START, sender=node04/default, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[node04/default]}, pendingCH=null, throwable=null, viewId=4}: java.lang.NullPointerException
> at org.infinispan.distribution.ch.DefaultConsistentHash.union(DefaultConsistentHash.java:243) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.union(DefaultConsistentHashFactory.java:120) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.union(DefaultConsistentHashFactory.java:45) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:168) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:253) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.JChannel.up(JChannel.java:707) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.RSVP.up(RSVP.java:172) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_37]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_37]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_37]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-2713) REBALANCE_START and REBALANCE_CONFIRM commands deadlock when RSVP.ack_on_delivery=true
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2713?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2713:
--------------------------------
Labels: nbst (was: )
> REBALANCE_START and REBALANCE_CONFIRM commands deadlock when RSVP.ack_on_delivery=true
> --------------------------------------------------------------------------------------
>
> Key: ISPN-2713
> URL: https://issues.jboss.org/browse/ISPN-2713
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: nbst
> Fix For: 5.3.0.Beta1, 5.3.0.Final
>
>
> When the coordinator sends a REBALANCE_START command, it holds a lock on the ClusterCacheStatus until it receives the responses from all the other members.
> If a node doesn't need to request any new state, it sends the rebalance confirmation to the coordinator on the same thread that received the REBALANCE_START command. The REBALANCE_CONFIRM command also wants to acquire a lock on the ClusterCacheStatus on the coordinator, but because the REBALANCE_CONFIRM command is sent asynchronously, it doesn't deadlock with the thread waiting for REBALANCE_START responses on the coordinator.
> At least, that's what happens when {{RSVP.ack_on_delivery=false}} (the Infinispan default). When {{RSVP.ack_on_delivery=true}} (the JGroups default), the "asynchronous" REBALANCE_CONFIRM command becomes synchronous, and it generates a deadlock. The rebalance then fails after the RSVP timeout expires (10 seconds by default).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-3129) If the status recovery fails for a cache, it stops recovery for all the caches
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3129?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3129:
--------------------------------
Labels: nbst (was: )
> If the status recovery fails for a cache, it stops recovery for all the caches
> ------------------------------------------------------------------------------
>
> Key: ISPN-3129
> URL: https://issues.jboss.org/browse/ISPN-3129
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.6.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: nbst
> Fix For: 5.2.7.Final, 5.3.0.CR1, 5.3.0.Final
>
>
> When a node becomes coordinator, it sends a GET_STATUS command to recover the status of all the caches in the cluster. Then it sends a CH_UPDATE command for each cache, to install a new topology.
> However, if the CH_UPDATE command fails for one cache, the coordinator will interrupt the recovery process, and it won't even install the cache topology for the rest of the caches locally. When a new node joins, it will act as if the cache was just started:
> {code}
> 09:05:22,542 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-1693,shared=udp) Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=default-host/clusterbench-granular, type=CH_UPDATE, sender=perf03/web, joinInfo=null, topologyId=21, currentCH=DefaultConsistentHash{numSegments=80, numOwners=2, members=[perf04/web, perf01/web, perf02/web]}, pendingCH=null, throwable=null, viewId=7} [sender=perf03/web]
> 09:06:21,484 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-2231,shared=udp) Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=null, type=GET_STATUS, sender=perf04/web, joinInfo=null, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=7} [sender=perf04/web]
> 09:28:21,060 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-22206,shared=udp) Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=default-host/clusterbench-granular, type=CH_UPDATE, sender=perf04/web, joinInfo=null, topologyId=1, currentCH=DefaultConsistentHash{numSegments=80, numOwners=2, members=[perf04/web, perf02/web, perf03/web]}, pendingCH=null, throwable=null, viewId=10} [sender=perf04/web]
> {code}
> This is mitigated by the ISPN-2966 fix (https://github.com/infinispan/infinispan/pull/1742). Since the CH_UPDATE command is now sent asynchronously, it ignores any exceptions on the targets. But on the 5.2.x branch, the CH_UPDATE command is sent synchronously, and it can fail with an exception like this:
> {code}
> 05:43:56,513 WARN [org.infinispan.topology.CacheTopologyControlCommand] (notification-thread-0) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=default-host/clusterbench, type=CH_UPDATE, sender=perf21/web, joinInfo=null, topologyId=23, currentCH=DefaultConsistentHash{numSegments=80, numOwners=2, members=[perf21/web, perf18/web, perf19/web]}, pendingCH=null, throwable=null, viewId=8}: java.lang.IllegalArgumentException: The task is already cancelled.
> at org.infinispan.statetransfer.InboundTransferTask.cancelSegments(InboundTransferTask.java:167)
> at org.infinispan.statetransfer.StateConsumerImpl.cancelTransfers(StateConsumerImpl.java:774)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:314)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:195)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:61)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:121)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:202)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:165)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137)
> at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastConsistentHashUpdate(ClusterTopologyManagerImpl.java:332)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheStatusAfterMerge(ClusterTopologyManagerImpl.java:319)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleNewView(ClusterTopologyManagerImpl.java:236)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.handleViewChange(ClusterTopologyManagerImpl.java:579)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_43]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_43]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_43]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_43]
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:212)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
> {code}
> If that happens, cluster recovery is interrupted and some caches may end up in a corrupted state where it's impossible for new members to join.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-2935) After view change single node is left in cluster, but still tries to invoke stuff remotely
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2935?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2935:
--------------------------------
Labels: nbst (was: )
> After view change single node is left in cluster, but still tries to invoke stuff remotely
> ------------------------------------------------------------------------------------------
>
> Key: ISPN-2935
> URL: https://issues.jboss.org/browse/ISPN-2935
> Project: Infinispan
> Issue Type: Bug
> Components: RPC
> Affects Versions: 5.2.4.Final
> Reporter: Thomas Fromm
> Assignee: Dan Berindei
> Priority: Critical
> Labels: nbst
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: jgroups-udp.xml, suspect.log
>
>
> Cluster size shrinks down to one single node.
> The situation clears up when a second node joins. Until this point the single node is not usable, since mostly all operations fail.
> Unfortunality its very hard to reproduce and I do not have TRACE logs for that :-/
> {code}
> WARN 18.03.13 22:57:45,576 [Incoming-4,IBIS-24629] ISPNCacheManager viewChanged: EventImpl{type=VIEW_CHANGED, newMembers=[IBIS-18850, IBIS-24629], oldMembers=[IBIS-18850, IBIS-9207, IBIS-24629], localAddress=IBIS-24629, viewId=43, subgroupsMerged=null, mergeView=false}
> WARN 18.03.13 22:58:04,784 [Incoming-4,IBIS-24629] ISPNCacheManager viewChanged: EventImpl{type=VIEW_CHANGED, newMembers=[IBIS-24629], oldMembers=[IBIS-18850, IBIS-24629], localAddress=IBIS-24629, viewId=44, subgroupsMerged=null, mergeView=false}
> ERROR 18.03.13 22:58:15,230 [IS-dev-blade-Node-8-WThread-4] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='benchmark_mux212072522214483374'}, value=ISPNCacheValue{value=key=benchmark_mux212072522214483374, startTime=Mon Mar 18 23:58:15 CET 2013, workflow=benchmark_scenario_2, module = benchmark_mux, modified=1363643895223}, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-8371768971330441521, lockIntention=null, affectedKeys=[CacheKey{name='benchmark_mux212072522214483374'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232739:local, cacheName='EXEC_TIMEOUT', topologyId=127}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> ...
>
> Its ok when TX during view changes fails but....
>
> ERROR 18.03.13 23:02:38,813 [http-bio-8000-exec-72] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:1035005977 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644158809}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,815 [http-bio-8000-exec-72] TransactionCoordinator ISPN000097: Error while processing a prepare in a single-phase transaction
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:1035005977 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644158809}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,815 [http-bio-8000-exec-72] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> WARN 18.03.13 23:02:38,816 [http-bio-8000-exec-72] TransactionCoordinator ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf81
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,817 [http-bio-8000-exec-72] DummyTransaction ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf81} org.infinispan.transaction.synchronization.SynchronizationAdapter@12cfa0
> org.infinispan.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:220)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> ... 32 more
> {code}
>
> ... and more and more... even much later... far beyond any TX timeouts...
> {code}
> ...
> WARN 18.03.13 23:07:55,844 [Timer-3,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> WARN 18.03.13 23:08:01,595 [Timer-1,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> WARN 18.03.13 23:08:07,684 [Timer-2,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> {code}
> ...
> {code}
> ERROR 18.03.13 23:09:21,944 [http-bio-8000-exec-141] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:641451160 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644561942}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,945 [http-bio-8000-exec-141] TransactionCoordinator ISPN000097: Error while processing a prepare in a single-phase transaction
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:641451160 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644561942}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,945 [http-bio-8000-exec-141] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> WARN 18.03.13 23:09:21,946 [http-bio-8000-exec-141] TransactionCoordinator ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf83
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,946 [http-bio-8000-exec-141] DummyTransaction ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf83} org.infinispan.transaction.synchronization.SynchronizationAdapter@12cfa2
> org.infinispan.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:220)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> ... 32 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-2910) Divide by zero exception on shutdown
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2910?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2910:
--------------------------------
Labels: nbst (was: )
> Divide by zero exception on shutdown
> ------------------------------------
>
> Key: ISPN-2910
> URL: https://issues.jboss.org/browse/ISPN-2910
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.1.Final
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
> Labels: nbst
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> 19:40:50,671 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,shared=udp) ISPN000094: Received new cluster view: [perf20/web|13] [perf20/web, perf21/web]
> 19:40:50,754 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (notification-thread-0) ISPN000196: Failed to recover cluster state after the current node became the coordinator: java.lang.ArithmeticException: / by zero
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.addPrimaryOwners(DefaultConsistentHashFactory.java:130)
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalanceBuilder(DefaultConsistentHashFactory.java:124)
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.updateMembers(DefaultConsistentHashFactory.java:86)
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.updateMembers(DefaultConsistentHashFactory.java:45)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheStatusAfterMerge(ClusterTopologyManagerImpl.java:306)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleNewView(ClusterTopologyManagerImpl.java:236)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.handleViewChange(ClusterTopologyManagerImpl.java:597)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_38]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_38]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_38]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_38]
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:212)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-2825) ClusterTopologyManagerImpl should not hold a lock while invoking an RPC
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2825?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2825:
--------------------------------
Labels: nbst (was: )
> ClusterTopologyManagerImpl should not hold a lock while invoking an RPC
> -----------------------------------------------------------------------
>
> Key: ISPN-2825
> URL: https://issues.jboss.org/browse/ISPN-2825
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: nbst
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> On the coordinator, ClusterTopologyManagerImpl holds a lock on a cache's ClusterCacheStatus while it is invoking a synchronous REBALANCE_START or CH_UPDATE command. This helps ensure the ordering of the commands is the same on all the members.
> However, this has some downsides. On a joining node, it takes quite some time before replying to the coordinator (as it needs to request transactions from the other nodes). The nodes that don't need to request any data will send a REBALANCE_CONFIRM command to the coordinator right away, but that command will block on the ClusterCacheStatus lock. If the number of OOB threads is limited, this can even lead to a deadlock.
> Now that CH_UPDATE commands also increment the topology id, we don't really need to enforce the same ordering. If a CH_UPDATE command is sent after a REBALANCE_START command but arrives before it, LocalTopologyManagerImpl just needs to act as if the CH_UPDATE command was actually a REBALANCE_START. (It knows there should be a rebalance when a CH_UPDATE command has pendingCH != null.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-2918) TopologyAwareConsistentHashFactory doesn't distribute data to nodes evenly
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2918?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2918:
--------------------------------
Labels: 5.2.x jdg6 nbst (was: 5.2.x jdg6)
> TopologyAwareConsistentHashFactory doesn't distribute data to nodes evenly
> --------------------------------------------------------------------------
>
> Key: ISPN-2918
> URL: https://issues.jboss.org/browse/ISPN-2918
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x, jdg6, nbst
> Fix For: 5.2.6.Final, 5.3.0.Alpha1
>
>
> When the topology of a cluster is "balanced" (i.e. all sites have the same number of racks, racks have the same number of machines, machines have the same number of nodes), the number of segments owned by each node should be approximately the same.
> The current algorithm properly balances the primary owners of the segments, but the backup owners are not balanced, so a node can end up owning a lot more segments than expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
- 1
- 0