[JBoss JIRA] (ISPN-2931) async mode changes remove behaviour
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2931?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2931:
--------------------------------
Assignee: Galder Zamarreño (was: Mircea Markus)
> async mode changes remove behaviour
> -----------------------------------
>
> Key: ISPN-2931
> URL: https://issues.jboss.org/browse/ISPN-2931
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Galder Zamarreño
> Fix For: 5.3.0.Final
>
>
> With a cache setup as clustering dist, 2 owners and async mode the Cache.remove API does not behave correctly. Cache.remove(key) should return the old value and Cache.remove(key, value) should return true if the entry was removed. Both methods only work correctly if invoked on the primary owner of the key. If invoked on another node remove(key) returns null every time and remove(key,value) returns false every time. The Infinispan documentation says that in async mode these operations should work as expected. https://docs.jboss.org/author/display/ISPN/Asynchronous+Options
> Complete cache config:
> <namedCache name="distributed">
> <!-- Used to register JMX statistics in any available MBean server -->
> <jmxStatistics enabled="true" />
>
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="true" timeout="20000" />
> <hash numOwners="2"/>
> <async/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
>
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="3600000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2931) async mode changes remove behaviour
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2931?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2931:
--------------------------------
Fix Version/s: 5.3.0.Final
> async mode changes remove behaviour
> -----------------------------------
>
> Key: ISPN-2931
> URL: https://issues.jboss.org/browse/ISPN-2931
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Mircea Markus
> Fix For: 5.3.0.Final
>
>
> With a cache setup as clustering dist, 2 owners and async mode the Cache.remove API does not behave correctly. Cache.remove(key) should return the old value and Cache.remove(key, value) should return true if the entry was removed. Both methods only work correctly if invoked on the primary owner of the key. If invoked on another node remove(key) returns null every time and remove(key,value) returns false every time. The Infinispan documentation says that in async mode these operations should work as expected. https://docs.jboss.org/author/display/ISPN/Asynchronous+Options
> Complete cache config:
> <namedCache name="distributed">
> <!-- Used to register JMX statistics in any available MBean server -->
> <jmxStatistics enabled="true" />
>
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="true" timeout="20000" />
> <hash numOwners="2"/>
> <async/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
>
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="3600000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2972) REST server: the cache manager should not be singleton
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2972?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-2972.
--------------------------------
Resolution: Duplicate Issue
Oops, duplicate of ISPN-2936.
> REST server: the cache manager should not be singleton
> ------------------------------------------------------
>
> Key: ISPN-2972
> URL: https://issues.jboss.org/browse/ISPN-2972
> Project: Infinispan
> Issue Type: Task
> Components: Server
> Affects Versions: 5.2.0.Final
> Reporter: Dan Berindei
> Assignee: Michal Linhard
> Fix For: 5.3.0.Alpha1
>
>
> Holding the cache manager in a singleton hurts testability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2972) REST server: the cache manager should not be singleton
by Dan Berindei (JIRA)
Dan Berindei created ISPN-2972:
----------------------------------
Summary: REST server: the cache manager should not be singleton
Key: ISPN-2972
URL: https://issues.jboss.org/browse/ISPN-2972
Project: Infinispan
Issue Type: Task
Components: Server
Affects Versions: 5.2.0.Final
Reporter: Dan Berindei
Assignee: Michal Linhard
Fix For: 5.3.0.Alpha1
Holding the cache manager in a singleton hurts testability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2968) bdbje - fail to start clustered
by Aleksandar Kostadinov (JIRA)
[ https://issues.jboss.org/browse/ISPN-2968?page=com.atlassian.jira.plugin.... ]
Aleksandar Kostadinov commented on ISPN-2968:
---------------------------------------------
Is this the kind of info you are requesting:{code}
BasicCache<String, Object> users = provider.getCacheContainer().getCache("userCache");
BasicCache<PostKey, Object> posts = provider.getCacheContainer().getCache("postCache");
if (users.get("user1") != null) {
log.info("chunchun cache non-empty, skipping initialization");
return;
}
... {code}
In normal operation users.get("user1") would be non-null because first server has already initialized cache with test data. I don't do any put() to get the exception.
wrt logs, do you want tracing of whole org.infinispan or something more fine grained? Please let me know and I can deliver.
wrt test case, my guess is that any distributed cache test case you already have with bdbje cache store would fail the same. But I have no knowledge about infinispan test suite to write a test case.
> bdbje - fail to start clustered
> -------------------------------
>
> Key: ISPN-2968
> URL: https://issues.jboss.org/browse/ISPN-2968
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 5.2.5.Final
> Environment: fedora 18, openjdk 1.7
> Reporter: Aleksandar Kostadinov
> Assignee: Mircea Markus
> Labels: bdbje, clustered
> Fix For: 6.0.0.Final
>
> Attachments: infinispan-config.xml
>
>
> with bdbje store I can start a server and everything seems to be working fine. Without a cache store I can start a cluster of two servers and everything seems to be working fine. But having configured bdbje backend and trying to start a second server serults in exceptions like these:
> {code}22:55:15,479 ERROR [org.infinispan.transaction.TransactionCoordinator] (OOB-11,koTapaH-7896) Error while processing prepare: org.infinispan.CacheException: org.infinispan.loaders.CacheLoaderException: Problem preparing transaction
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:351) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
> at org.infinispan.statetransfer.StateConsumerImpl.doApplyState(StateConsumerImpl.java:488) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:432) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:86) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:472) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.JChannel.up(JChannel.java:719) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1008) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.RSVP.up(RSVP.java:163) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:161) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:404) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:386) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:893) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:790) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:411) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:609) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:286) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:206) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.Discovery.up(Discovery.java:375) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1350) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1491) [jgroups-3.3.0.CR1.jar:3.3.0.CR1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea]
> at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
> Caused by: org.infinispan.loaders.CacheLoaderException: Problem preparing transaction
> at org.infinispan.loaders.bdbje.BdbjeCacheStore.convertToCacheLoaderException(BdbjeCacheStore.java:589) [infinispan-cachestore-bdbje-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.loaders.bdbje.BdbjeCacheStore.prepare(BdbjeCacheStore.java:315) [infinispan-cachestore-bdbje-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.loaders.bdbje.BdbjeCacheStore.prepare(BdbjeCacheStore.java:274) [infinispan-cachestore-bdbje-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.CacheStoreInterceptor.prepareCacheLoader(CacheStoreInterceptor.java:303) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:158) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.5.Final.jar:5.2.5.Final]
> ... 43 more
> Caused by: java.io.NotSerializableException: org.infinispan.io.ImmutableMarshalledValueByteStream
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1180) [rt.jar:1.7.0_09-icedtea]
> at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528) [rt.jar:1.7.0_09-icedtea]
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493) [rt.jar:1.7.0_09-icedtea]
> at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) [rt.jar:1.7.0_09-icedtea]
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) [rt.jar:1.7.0_09-icedtea]
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346) [rt.jar:1.7.0_09-icedtea]
> at com.sleepycat.bind.serial.SerialBinding.objectToEntry(SerialBinding.java:179) [je-5.0.34.jar:5.0.34]
> at com.sleepycat.collections.DataView.useKey(DataView.java:502) [je-5.0.34.jar:5.0.34]
> at com.sleepycat.collections.DataCursor.initForPut(DataCursor.java:805) [je-5.0.34.jar:5.0.34]
> at com.sleepycat.collections.DataCursor.put(DataCursor.java:751) [je-5.0.34.jar:5.0.34]
> at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:321) [je-5.0.34.jar:5.0.34]
> at com.sleepycat.collections.StoredMap.put(StoredMap.java:279) [je-5.0.34.jar:5.0.34]
> at org.infinispan.loaders.bdbje.BdbjeCacheStore.store(BdbjeCacheStore.java:429) [infinispan-cachestore-bdbje-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.loaders.bdbje.ModificationsTransactionWorker.doWork(ModificationsTransactionWorker.java:66) [infinispan-cachestore-bdbje-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.loaders.bdbje.PreparableTransactionRunner.prepare(PreparableTransactionRunner.java:78) [infinispan-cachestore-bdbje-5.2.5.Final.jar:5.2.5.Final]
> at org.infinispan.loaders.bdbje.BdbjeCacheStore.prepare(BdbjeCacheStore.java:309) [infinispan-cachestore-bdbje-5.2.5.Final.jar:5.2.5.Final]
> ... 93 more{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2935) After view change single node is left in cluster, but still tries to invoke stuff remotely
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2935?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2935:
------------------------------------
I think the proper fix is a little more complicated, not all the cluster members are members of the cache. We also need to synchronize on the {{viewUpdateLock}} in {{ClusterTopologyManagerImpl.updateCacheStatusAfterMerge()}} to ensure an ordering between the topology created when the node becomes coordinator and the topology created when another node leaves.
I think we should be able to reproduce this with two leavers, if we delay both the view delivery to ClusterTopologyManagerImpl (with a blocking view listener) and the GET_STATUS reply from the remaining node(s).
> After view change single node is left in cluster, but still tries to invoke stuff remotely
> ------------------------------------------------------------------------------------------
>
> Key: ISPN-2935
> URL: https://issues.jboss.org/browse/ISPN-2935
> Project: Infinispan
> Issue Type: Bug
> Components: RPC
> Affects Versions: 5.2.4.Final
> Reporter: Thomas Fromm
> Assignee: Adrian Nistor
> Priority: Critical
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: jgroups-udp.xml, suspect.log
>
>
> Cluster size shrinks down to one single node.
> The situation clears up when a second node joins. Until this point the single node is not usable, since mostly all operations fail.
> Unfortunality its very hard to reproduce and I do not have TRACE logs for that :-/
> {code}
> WARN 18.03.13 22:57:45,576 [Incoming-4,IBIS-24629] ISPNCacheManager viewChanged: EventImpl{type=VIEW_CHANGED, newMembers=[IBIS-18850, IBIS-24629], oldMembers=[IBIS-18850, IBIS-9207, IBIS-24629], localAddress=IBIS-24629, viewId=43, subgroupsMerged=null, mergeView=false}
> WARN 18.03.13 22:58:04,784 [Incoming-4,IBIS-24629] ISPNCacheManager viewChanged: EventImpl{type=VIEW_CHANGED, newMembers=[IBIS-24629], oldMembers=[IBIS-18850, IBIS-24629], localAddress=IBIS-24629, viewId=44, subgroupsMerged=null, mergeView=false}
> ERROR 18.03.13 22:58:15,230 [IS-dev-blade-Node-8-WThread-4] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='benchmark_mux212072522214483374'}, value=ISPNCacheValue{value=key=benchmark_mux212072522214483374, startTime=Mon Mar 18 23:58:15 CET 2013, workflow=benchmark_scenario_2, module = benchmark_mux, modified=1363643895223}, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-8371768971330441521, lockIntention=null, affectedKeys=[CacheKey{name='benchmark_mux212072522214483374'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232739:local, cacheName='EXEC_TIMEOUT', topologyId=127}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> ...
>
> Its ok when TX during view changes fails but....
>
> ERROR 18.03.13 23:02:38,813 [http-bio-8000-exec-72] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:1035005977 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644158809}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,815 [http-bio-8000-exec-72] TransactionCoordinator ISPN000097: Error while processing a prepare in a single-phase transaction
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:1035005977 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644158809}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,815 [http-bio-8000-exec-72] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> WARN 18.03.13 23:02:38,816 [http-bio-8000-exec-72] TransactionCoordinator ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf81
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,817 [http-bio-8000-exec-72] DummyTransaction ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf81} org.infinispan.transaction.synchronization.SynchronizationAdapter@12cfa0
> org.infinispan.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:220)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> ... 32 more
> {code}
>
> ... and more and more... even much later... far beyond any TX timeouts...
> {code}
> ...
> WARN 18.03.13 23:07:55,844 [Timer-3,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> WARN 18.03.13 23:08:01,595 [Timer-1,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> WARN 18.03.13 23:08:07,684 [Timer-2,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> {code}
> ...
> {code}
> ERROR 18.03.13 23:09:21,944 [http-bio-8000-exec-141] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:641451160 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644561942}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,945 [http-bio-8000-exec-141] TransactionCoordinator ISPN000097: Error while processing a prepare in a single-phase transaction
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:641451160 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644561942}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,945 [http-bio-8000-exec-141] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> WARN 18.03.13 23:09:21,946 [http-bio-8000-exec-141] TransactionCoordinator ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf83
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,946 [http-bio-8000-exec-141] DummyTransaction ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf83} org.infinispan.transaction.synchronization.SynchronizationAdapter@12cfa2
> org.infinispan.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:220)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> ... 32 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2939) Remove option for async Marshalling
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2939?page=com.atlassian.jira.plugin.... ]
Mircea Markus resolved ISPN-2939.
---------------------------------
Resolution: Rejected
the async marshalling still has its performance benefits, when the number of writes is small.
> Remove option for async Marshalling
> -----------------------------------
>
> Key: ISPN-2939
> URL: https://issues.jboss.org/browse/ISPN-2939
> Project: Infinispan
> Issue Type: Task
> Components: Core API
> Reporter: Sanne Grinovero
> Assignee: Mircea Markus
> Priority: Minor
> Fix For: 6.0.0.Beta1
>
>
> Seems to introduce inconsistencies - and generally hard to figure out how to configure properly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2948) ConcurrentModificationException in TransactionCoordinator.commit()
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2948?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-2948:
-------------------------------------
I'm rejecting this as it is not related to infinispan: an external object that is kept in Infinispan throws an exception during serialization, because of concurrent modifications.
> ConcurrentModificationException in TransactionCoordinator.commit()
> ------------------------------------------------------------------
>
> Key: ISPN-2948
> URL: https://issues.jboss.org/browse/ISPN-2948
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.1.Final
> Reporter: Clement Pang
> Assignee: Mircea Markus
>
> Not entirely sure how to reproduce this but it seems like a map is modified while it is being serialized.
> {code}
> 11:19:29,584 ERROR org.infinispan.interceptors.InvocationContextInterceptor (http-/0.0.0.0:8080-2) ISPN000136: Execution error: java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:390) rt.jar:1.6.0_27
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:409) rt.jar:1.6.0_27
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:408) rt.jar:1.6.0_27
> at java.util.HashMap.writeObject(HashMap.java:1016) rt.jar:1.6.0_27
> at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source) :1.6.0_27
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) rt.jar:1.6.0_27
> at java.lang.reflect.Method.invoke(Method.java:616) rt.jar:1.6.0_27
> at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:180)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1007)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:680)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.jboss.as.clustering.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:85)
> at org.jboss.as.clustering.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:175)
> at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:47)
> at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:36)
> at org.infinispan.marshall.jboss.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:457) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.marshall.MarshallUtil.marshallMap(MarshallUtil.java:59) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:63) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:47) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:237) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:234) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:154) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:150) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.loaders.bucket.Bucket$Externalizer.writeObject(Bucket.java:148) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.Bucket$Externalizer.writeObject(Bucket.java:139) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:94) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:84) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToByteBuffer(AbstractDelegatingMarshaller.java:94) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:335) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:89) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.AbstractCacheStore.applyModifications(AbstractCacheStore.java:126) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.AbstractCacheStore.commit(AbstractCacheStore.java:163) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.CacheStoreInterceptor.commitCommand(CacheStoreInterceptor.java:164) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.CacheStoreInterceptor.visitCommitCommand(CacheStoreInterceptor.java:146) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:116) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:85) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
> at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1527) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:142) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at java.lang.Thread.run(Thread.java:679) rt.jar:1.6.0_27
> Caused by: an exception which occurred:
> in object java.util.HashMap@c8c53a7f
> in object org.jboss.as.clustering.SimpleMarshalledValue@c8c53a7f
> in object org.infinispan.util.FastCopyHashMap@6685286f
> in object org.infinispan.atomic.AtomicHashMap@5dd18547
> in object org.infinispan.container.entries.ImmortalCacheEntry@a8322011
> in object org.infinispan.loaders.bucket.Bucket@31cbb8e4
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2948) ConcurrentModificationException in TransactionCoordinator.commit()
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2948?page=com.atlassian.jira.plugin.... ]
Mircea Markus resolved ISPN-2948.
---------------------------------
Resolution: Rejected
see last comment.
> ConcurrentModificationException in TransactionCoordinator.commit()
> ------------------------------------------------------------------
>
> Key: ISPN-2948
> URL: https://issues.jboss.org/browse/ISPN-2948
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.1.Final
> Reporter: Clement Pang
> Assignee: Mircea Markus
>
> Not entirely sure how to reproduce this but it seems like a map is modified while it is being serialized.
> {code}
> 11:19:29,584 ERROR org.infinispan.interceptors.InvocationContextInterceptor (http-/0.0.0.0:8080-2) ISPN000136: Execution error: java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:390) rt.jar:1.6.0_27
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:409) rt.jar:1.6.0_27
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:408) rt.jar:1.6.0_27
> at java.util.HashMap.writeObject(HashMap.java:1016) rt.jar:1.6.0_27
> at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source) :1.6.0_27
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) rt.jar:1.6.0_27
> at java.lang.reflect.Method.invoke(Method.java:616) rt.jar:1.6.0_27
> at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:180)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1007)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:680)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.jboss.as.clustering.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:85)
> at org.jboss.as.clustering.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:175)
> at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:47)
> at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:36)
> at org.infinispan.marshall.jboss.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:457) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.marshall.MarshallUtil.marshallMap(MarshallUtil.java:59) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:63) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:47) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:237) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:234) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:154) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:150) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.loaders.bucket.Bucket$Externalizer.writeObject(Bucket.java:148) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.Bucket$Externalizer.writeObject(Bucket.java:139) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:94) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:84) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToByteBuffer(AbstractDelegatingMarshaller.java:94) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:335) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:89) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.AbstractCacheStore.applyModifications(AbstractCacheStore.java:126) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.AbstractCacheStore.commit(AbstractCacheStore.java:163) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.CacheStoreInterceptor.commitCommand(CacheStoreInterceptor.java:164) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.CacheStoreInterceptor.visitCommitCommand(CacheStoreInterceptor.java:146) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:116) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:85) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
> at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1527) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:142) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at java.lang.Thread.run(Thread.java:679) rt.jar:1.6.0_27
> Caused by: an exception which occurred:
> in object java.util.HashMap@c8c53a7f
> in object org.jboss.as.clustering.SimpleMarshalledValue@c8c53a7f
> in object org.infinispan.util.FastCopyHashMap@6685286f
> in object org.infinispan.atomic.AtomicHashMap@5dd18547
> in object org.infinispan.container.entries.ImmortalCacheEntry@a8322011
> in object org.infinispan.loaders.bucket.Bucket@31cbb8e4
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0
[JBoss JIRA] (ISPN-2477) AsyncStore shutdown can leak threads
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2477?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2477:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1736
This first pull request fixes the issue only partially. Please leave it open for now.
> AsyncStore shutdown can leak threads
> ------------------------------------
>
> Key: ISPN-2477
> URL: https://issues.jboss.org/browse/ISPN-2477
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 5.2.0.Beta3
> Reporter: Adrian Nistor
> Assignee: Galder Zamarreño
> Fix For: 5.3.0.Final
>
>
> AsyncStore stop() should take care that all threads (coordinator and workers) are shut down. Right now there could be worker threads left hanging and the coordinator thread waiting forever for them to end.
> coordinator.join(shutdownTimeout) in AsyncStore.stop() could time out and both coordinator and worker thread will leak in this case.
> We must ensure the coordinator is terminated and the ExecutorService used for worker threads is also shut down when we exit AsyncStore.stop().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
- 1
- 0