[infinispan-issues] [JBoss JIRA] (ISPN-2935) After view change single node is left in cluster, but still tries to invoke stuff remotely

Tue Mar 19 07:01:45 EDT 2013

    [ https://issues.jboss.org/browse/ISPN-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761761#comment-12761761 ] 

Adrian Nistor commented on ISPN-2935:
-------------------------------------

This looks like ISPN-2402. It was there since long ago but now it reproduces more frequently because there was a harmful 'optimization' in RpcManager.invokeRemotely that was hiding this error and this was removed when fixing ISPN-2787. I think ISPN-2402 needs an urgent fix now.

It's weird this still happens long after the view has changed. We'll need a trace log to investigate this.

> After view change single node is left in cluster, but still tries to invoke stuff remotely
> ------------------------------------------------------------------------------------------
>
>                 Key: ISPN-2935
>                 URL: https://issues.jboss.org/browse/ISPN-2935
>             Project: Infinispan
>          Issue Type: Bug
>          Components: RPC
>    Affects Versions: 5.2.4.Final
>            Reporter: Thomas Fromm
>            Assignee: Mircea Markus
>            Priority: Critical
>         Attachments: jgroups-udp.xml
>
>
> Cluster size shrinks down to one single node.
> The situation clears up when a second node joins. Until this point the single node is not usable, since mostly all operations fail.
> Unfortunality its very hard to reproduce and I do not have TRACE logs for that :-/
> {code}
>  WARN 18.03.13 22:57:45,576 [Incoming-4,IBIS-24629] ISPNCacheManager          viewChanged: EventImpl{type=VIEW_CHANGED, newMembers=[IBIS-18850, IBIS-24629], oldMembers=[IBIS-18850, IBIS-9207, IBIS-24629], localAddress=IBIS-24629, viewId=43, subgroupsMerged=null, mergeView=false}
> WARN 18.03.13 22:58:04,784 [Incoming-4,IBIS-24629] ISPNCacheManager          viewChanged: EventImpl{type=VIEW_CHANGED, newMembers=[IBIS-24629], oldMembers=[IBIS-18850, IBIS-24629], localAddress=IBIS-24629, viewId=44, subgroupsMerged=null, mergeView=false}
> ERROR 18.03.13 22:58:15,230 [IS-dev-blade-Node-8-WThread-4] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='benchmark_mux212072522214483374'}, value=ISPNCacheValue{value=key=benchmark_mux212072522214483374, startTime=Mon Mar 18 23:58:15 CET 2013, workflow=benchmark_scenario_2, module = benchmark_mux, modified=1363643895223}, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-8371768971330441521, lockIntention=null, affectedKeys=[CacheKey{name='benchmark_mux212072522214483374'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232739:local, cacheName='EXEC_TIMEOUT', topologyId=127}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> ...
>  
> Its ok when TX during view changes fails but....
>  
> ERROR 18.03.13 23:02:38,813 [http-bio-8000-exec-72] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:1035005977 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644158809}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
>     at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
>     at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
>     at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,815 [http-bio-8000-exec-72] TransactionCoordinator    ISPN000097: Error while processing a prepare in a single-phase transaction
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:1035005977 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644158809}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
>     at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
>     at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
>     at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,815 [http-bio-8000-exec-72] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
>     at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
>     at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
>     at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> WARN 18.03.13 23:02:38,816 [http-bio-8000-exec-72] TransactionCoordinator    ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction at 12cf81
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
>     at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
>     at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
>     at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,817 [http-bio-8000-exec-72] DummyTransaction          ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction at 12cf81} org.infinispan.transaction.synchronization.SynchronizationAdapter at 12cfa0
> org.infinispan.CacheException: Could not commit.
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
>     at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:220)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     ... 32 more
>  {code}
>  
> ... and more and more... even much later... far beyond any TX timeouts...
>  {code}
> ...
> WARN 18.03.13 23:07:55,844 [Timer-3,IBIS-24629] UDP                      IBIS-24629: no physical address for IBIS-36674, dropping message
> WARN 18.03.13 23:08:01,595 [Timer-1,IBIS-24629] UDP                      IBIS-24629: no physical address for IBIS-36674, dropping message
> WARN 18.03.13 23:08:07,684 [Timer-2,IBIS-24629] UDP                      IBIS-24629: no physical address for IBIS-36674, dropping message
> {code}
> ...
> {code}
> ERROR 18.03.13 23:09:21,944 [http-bio-8000-exec-141] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:641451160 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644561942}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
>     at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
>     at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
>     at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,945 [http-bio-8000-exec-141] TransactionCoordinator    ISPN000097: Error while processing a prepare in a single-phase transaction
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:641451160 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644561942}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
>     at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
>     at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
>     at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
>     at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
>     at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
>     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,945 [http-bio-8000-exec-141] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
>     at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
>     at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
>     at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> WARN 18.03.13 23:09:21,946 [http-bio-8000-exec-141] TransactionCoordinator    ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction at 12cf83
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
>     at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
>     at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
>     at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
>     at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
>     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
>     at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
>     at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
>     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
>     at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
>     at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,946 [http-bio-8000-exec-141] DummyTransaction          ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction at 12cf83} org.infinispan.transaction.synchronization.SynchronizationAdapter at 12cfa2
> org.infinispan.CacheException: Could not commit.
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
>     at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
>     at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
>     at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
>     at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
>     at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
>     at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
>     at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
>     at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
>     at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
>     at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
>     at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>     at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
>     at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:220)
>     at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
>     at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
>     ... 32 more 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira