[JBoss JIRA] (ISPN-2935) After view change single node is left in cluster, but still tries to invoke stuff remotely
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2935?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2935:
------------------------------------
Thomas, you mean after it already shut down? If IBIS-36674 left before the first view you posted, at 22:57:45, surely it must be fully stopped at 23:07:55 :)
That probably means your cache didn't install a new topology since IBIS-36674 left, so the problem isn't related to only one node remaining in the cluster. I still think making the view listener asynchronous should fix it, though.
> After view change single node is left in cluster, but still tries to invoke stuff remotely
> ------------------------------------------------------------------------------------------
>
> Key: ISPN-2935
> URL: https://issues.jboss.org/browse/ISPN-2935
> Project: Infinispan
> Issue Type: Bug
> Components: RPC
> Affects Versions: 5.2.4.Final
> Reporter: Thomas Fromm
> Assignee: Adrian Nistor
> Priority: Critical
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: jgroups-udp.xml
>
>
> Cluster size shrinks down to one single node.
> The situation clears up when a second node joins. Until this point the single node is not usable, since mostly all operations fail.
> Unfortunality its very hard to reproduce and I do not have TRACE logs for that :-/
> {code}
> WARN 18.03.13 22:57:45,576 [Incoming-4,IBIS-24629] ISPNCacheManager viewChanged: EventImpl{type=VIEW_CHANGED, newMembers=[IBIS-18850, IBIS-24629], oldMembers=[IBIS-18850, IBIS-9207, IBIS-24629], localAddress=IBIS-24629, viewId=43, subgroupsMerged=null, mergeView=false}
> WARN 18.03.13 22:58:04,784 [Incoming-4,IBIS-24629] ISPNCacheManager viewChanged: EventImpl{type=VIEW_CHANGED, newMembers=[IBIS-24629], oldMembers=[IBIS-18850, IBIS-24629], localAddress=IBIS-24629, viewId=44, subgroupsMerged=null, mergeView=false}
> ERROR 18.03.13 22:58:15,230 [IS-dev-blade-Node-8-WThread-4] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='benchmark_mux212072522214483374'}, value=ISPNCacheValue{value=key=benchmark_mux212072522214483374, startTime=Mon Mar 18 23:58:15 CET 2013, workflow=benchmark_scenario_2, module = benchmark_mux, modified=1363643895223}, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-8371768971330441521, lockIntention=null, affectedKeys=[CacheKey{name='benchmark_mux212072522214483374'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232739:local, cacheName='EXEC_TIMEOUT', topologyId=127}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> ...
>
> Its ok when TX during view changes fails but....
>
> ERROR 18.03.13 23:02:38,813 [http-bio-8000-exec-72] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:1035005977 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644158809}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,815 [http-bio-8000-exec-72] TransactionCoordinator ISPN000097: Error while processing a prepare in a single-phase transaction
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:1035005977 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644158809}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,815 [http-bio-8000-exec-72] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> WARN 18.03.13 23:02:38,816 [http-bio-8000-exec-72] TransactionCoordinator ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf81
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=5804954807762587563, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232769:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:02:38,817 [http-bio-8000-exec-72] DummyTransaction ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf81} org.infinispan.transaction.synchronization.SynchronizationAdapter@12cfa0
> org.infinispan.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:220)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> ... 32 more
> {code}
>
> ... and more and more... even much later... far beyond any TX timeouts...
> {code}
> ...
> WARN 18.03.13 23:07:55,844 [Timer-3,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> WARN 18.03.13 23:08:01,595 [Timer-1,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> WARN 18.03.13 23:08:07,684 [Timer-2,IBIS-24629] UDP IBIS-24629: no physical address for IBIS-36674, dropping message
> {code}
> ...
> {code}
> ERROR 18.03.13 23:09:21,944 [http-bio-8000-exec-141] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:641451160 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644561942}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,945 [http-bio-8000-exec-141] TransactionCoordinator ISPN000097: Error while processing a prepare in a single-phase transaction
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command PrepareCommand {modifications=[PutKeyValueCommand{key=CacheKey{name='ProcessId'}, value=ISPNCacheValue{value=
> Property:ProcessId Class:com.inubit.ibis.utils.property.type.LongProperty Hash:641451160 Type:class java.lang.Long Deprecated:false New:false RawDisplayName:null
> Value:14540000, modified=1363644561942}, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}], onePhaseCommit=true, gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPrepareCommand(DeadlockDetectingInterceptor.java:107)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPrepareCommand(DistCacheStoreInterceptor.java:160)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:107)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,945 [http-bio-8000-exec-141] InvocationContextInterceptor ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> WARN 18.03.13 23:09:21,946 [http-bio-8000-exec-141] TransactionCoordinator ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf83
> org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=DldGlobalTransaction{coinToss=-6030295490446405650, lockIntention=null, affectedKeys=[CacheKey{name='ProcessId'}], locksAtOrigin=[]} GlobalTransaction:<IBIS-24629>:1232771:local, cacheName='RUNTIME_PROPERTIES', topologyId=129}
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:482)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:255)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheStoreInterceptor.visitRollbackCommand(CacheStoreInterceptor.java:202)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:67)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:72)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:167)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:85)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:126)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:131)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:61)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:232)
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> ERROR 18.03.13 23:09:21,946 [http-bio-8000-exec-141] DummyTransaction ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=129, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@12cf83} org.infinispan.transaction.synchronization.SynchronizationAdapter@12cfa2
> org.infinispan.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
> at com.inubit.ibis.server.cache.ispn.ISPNCache.commit(ISPNCache.java:191)
> at com.inubit.ibis.server.datafacade.cache.LockedCacheEntryAction.lockAndPerform(LockedCacheEntryAction.java:101)
> at com.inubit.ibis.server.cache.CacheEntryBasedCounter.getNextId(CacheEntryBasedCounter.java:105)
> at com.inubit.ibis.server.datafacade.cache.ConfigurationDataCacheFacade.getNextProcessId(ConfigurationDataCacheFacade.java:364)
> at com.inubit.ibis.server.IBISController.generateNewWorkflowId(IBISController.java:1265)
> at com.inubit.ibis.server.IBISController.convertMessage(IBISController.java:3080)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2994)
> at com.inubit.ibis.server.IBISController.processMessage(IBISController.java:2862)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doPost(IBISHTTPUploadServlet.java:467)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.doGet(IBISHTTPUploadServlet.java:89)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at com.inubit.ibis.servlets.IBISHTTPUploadServlet.service(IBISHTTPUploadServlet.java:58)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:220)
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:177)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
> ... 32 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2918) TopologyAwareConsistentHashFactory doesn't distribute data to nodes evenly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2918?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2918:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 923928|https://bugzilla.redhat.com/show_bug.cgi?id=923928] from NEW to MODIFIED
> TopologyAwareConsistentHashFactory doesn't distribute data to nodes evenly
> --------------------------------------------------------------------------
>
> Key: ISPN-2918
> URL: https://issues.jboss.org/browse/ISPN-2918
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x, jdg6
> Fix For: 5.2.6.Final, 5.3.0.Alpha1
>
>
> When the topology of a cluster is "balanced" (i.e. all sites have the same number of racks, racks have the same number of machines, machines have the same number of nodes), the number of segments owned by each node should be approximately the same.
> The current algorithm properly balances the primary owners of the segments, but the backup owners are not balanced, so a node can end up owning a lot more segments than expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2948) ConcurrentModificationException in TransactionCoordinator.commit()
by Clement Pang (JIRA)
[ https://issues.jboss.org/browse/ISPN-2948?page=com.atlassian.jira.plugin.... ]
Clement Pang commented on ISPN-2948:
------------------------------------
This is infinispan running within JBossAS EAP 6.1.0.Alpha. Attached is the entire configuration subsection.
{code}
<subsystem xmlns="urn:jboss:domain:infinispan:1.4">
<cache-container name="web" aliases="standard-session-cache" default-cache="local-web"
module="org.jboss.as.clustering.web.infinispan">
<local-cache name="local-web" batching="true">
<file-store passivation="false" purge="false"/>
</local-cache>
</cache-container>
<cache-container name="hibernate" default-cache="local-query" module="org.jboss.as.jpa.hibernate:4">
<local-cache name="entity">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<local-cache name="local-query">
<transaction mode="NONE"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<local-cache name="timestamps">
<transaction mode="NONE"/>
<eviction strategy="NONE"/>
</local-cache>
</cache-container>
<cache-container name="modeshape">
<local-cache name="dshell">
<locking striping="false" isolation="READ_COMMITTED"/>
<transaction mode="FULL_XA" locking="PESSIMISTIC"/>
<file-store path="modeshape" passivation="false" purge="false">
<write-behind/>
</file-store>
</local-cache>
</cache-container>
<cache-container name="lucene">
<local-cache name="dshell-index-data" start="EAGER">
<file-store path="lucene" purge="false" passivation="false">
<write-behind/>
</file-store>
</local-cache>
<local-cache name="dshell-index-metadata" start="EAGER">
<file-store path="lucene" purge="false" passivation="false">
<write-behind/>
</file-store>
</local-cache>
<local-cache name="dshell-index-lock" start="EAGER"/>
</cache-container>
</subsystem>
{code}
> ConcurrentModificationException in TransactionCoordinator.commit()
> ------------------------------------------------------------------
>
> Key: ISPN-2948
> URL: https://issues.jboss.org/browse/ISPN-2948
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.1.Final
> Reporter: Clement Pang
> Assignee: Mircea Markus
>
> Not entirely sure how to reproduce this but it seems like a map is modified while it is being serialized.
> {code}
> 11:19:29,584 ERROR org.infinispan.interceptors.InvocationContextInterceptor (http-/0.0.0.0:8080-2) ISPN000136: Execution error: java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:390) rt.jar:1.6.0_27
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:409) rt.jar:1.6.0_27
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:408) rt.jar:1.6.0_27
> at java.util.HashMap.writeObject(HashMap.java:1016) rt.jar:1.6.0_27
> at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source) :1.6.0_27
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) rt.jar:1.6.0_27
> at java.lang.reflect.Method.invoke(Method.java:616) rt.jar:1.6.0_27
> at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:180)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1007)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:680)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.jboss.as.clustering.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:85)
> at org.jboss.as.clustering.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:175)
> at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:47)
> at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:36)
> at org.infinispan.marshall.jboss.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:457) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.marshall.MarshallUtil.marshallMap(MarshallUtil.java:59) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:63) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:47) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:237) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:234) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:154) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:150) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.loaders.bucket.Bucket$Externalizer.writeObject(Bucket.java:148) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.Bucket$Externalizer.writeObject(Bucket.java:139) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
> at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:94) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:84) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToByteBuffer(AbstractDelegatingMarshaller.java:94) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:335) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:89) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.AbstractCacheStore.applyModifications(AbstractCacheStore.java:126) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.loaders.AbstractCacheStore.commit(AbstractCacheStore.java:163) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.CacheStoreInterceptor.commitCommand(CacheStoreInterceptor.java:164) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.CacheStoreInterceptor.visitCommitCommand(CacheStoreInterceptor.java:146) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:116) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:85) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) infinispan-core-5.2.1.Final.jar:5.2.1.Final
> at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
> at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1527) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:142) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) jbossweb-7.2.0.Final.jar:7.2.0.Final
> at java.lang.Thread.run(Thread.java:679) rt.jar:1.6.0_27
> Caused by: an exception which occurred:
> in object java.util.HashMap@c8c53a7f
> in object org.jboss.as.clustering.SimpleMarshalledValue@c8c53a7f
> in object org.infinispan.util.FastCopyHashMap@6685286f
> in object org.infinispan.atomic.AtomicHashMap@5dd18547
> in object org.infinispan.container.entries.ImmortalCacheEntry@a8322011
> in object org.infinispan.loaders.bucket.Bucket@31cbb8e4
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2918) TopologyAwareConsistentHashFactory doesn't distribute data to nodes evenly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2918?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-2918:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=923928
> TopologyAwareConsistentHashFactory doesn't distribute data to nodes evenly
> --------------------------------------------------------------------------
>
> Key: ISPN-2918
> URL: https://issues.jboss.org/browse/ISPN-2918
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x, jdg6
> Fix For: 5.2.6.Final, 5.3.0.Alpha1
>
>
> When the topology of a cluster is "balanced" (i.e. all sites have the same number of racks, racks have the same number of machines, machines have the same number of nodes), the number of segments owned by each node should be approximately the same.
> The current algorithm properly balances the primary owners of the segments, but the backup owners are not balanced, so a node can end up owning a lot more segments than expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2953) Alternate statistics MBeans to allow for a Hawt.io Infinispan plugin
by Manik Surtani (JIRA)
Manik Surtani created ISPN-2953:
-----------------------------------
Summary: Alternate statistics MBeans to allow for a Hawt.io Infinispan plugin
Key: ISPN-2953
URL: https://issues.jboss.org/browse/ISPN-2953
Project: Infinispan
Issue Type: Feature Request
Components: JMX, reporting and management
Affects Versions: 5.2.5.Final
Reporter: Manik Surtani
Assignee: Manik Surtani
Fix For: 5.3.0.Alpha1, 5.3.0.Final
Hawt.io draws its MBean trees in a manner different from what RHQ does, and the existing statistics MBean is hard to consume from Hawt.io.
This feature request is to register an alternate set of MBeans if a flag is passed in ({{-Dinfinispan.jmx.alternate_name_order=true}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2950) In distributed mode cache store data should be read through the main data owner (vs directly from the store)
by Manik Surtani (JIRA)
[ https://issues.jboss.org/browse/ISPN-2950?page=com.atlassian.jira.plugin.... ]
Manik Surtani commented on ISPN-2950:
-------------------------------------
Agreed, what's ClusterCacheLoader got to do with this issue? Pls update the description to remove that line.
Re: the fix itself, I think this should be scoped for 5.3. It's too late for the 5.2.x series. I think it is mostly a perf issue, it can be a consistency issue if using an async cache store (write behind), but that's how things have been for a while now and rushing in with a patch now for 5.2.x isn't the correct approach. Definitely critical for 5.3.0 IMO.
> In distributed mode cache store data should be read through the main data owner (vs directly from the store)
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2950
> URL: https://issues.jboss.org/browse/ISPN-2950
> Project: Infinispan
> Issue Type: Feature Request
> Components: Loaders and Stores
> Reporter: Mircea Markus
> Assignee: Mircea Markus
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> Dist cache with a cache store(shared or not), k owned by \{N1, N2\}. k is read on N3. What currently happens at this stage, if k is not present in N3's memory (likely unless L1 is configured), the N3's cache store is queried and data is loaded from there. This has several drawback:
> - the data might already be in the memory of the owner node (N1,N2) so reading it from the disk is highly inefficient. Especially for hot data: data requested from various nodes at the same time (see also mailing list discussion around lucene query performance depending on this)
> - if this is a local cache store, it might contain stale data which would be returned to the user
> - for async configured cache store this would result in dirty reads, given that a change might be in the async store's memory but not in the store at the moment when it is in read by N3. (Note that using async stores still leaves place to inconsistencies when a node leaves, e.g. because of node crashing before managing to flush the async store.)
> This JIRA is about changing the distribution mode: when asked for a specific key, a node would only touch a cache store if it is an owner of that key, otherwise would first go to the main owner of the key to read the value from there. The ClusterCacheLoader should be deprecated as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2918) TopologyAwareConsistentHashFactory doesn't distribute data to nodes evenly
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2918?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2918:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Incomplete Description
Integrated in master and 5.2.x. Thanks!
> TopologyAwareConsistentHashFactory doesn't distribute data to nodes evenly
> --------------------------------------------------------------------------
>
> Key: ISPN-2918
> URL: https://issues.jboss.org/browse/ISPN-2918
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x, jdg6
> Fix For: 5.2.6.Final, 5.3.0.Alpha1
>
>
> When the topology of a cluster is "balanced" (i.e. all sites have the same number of racks, racks have the same number of machines, machines have the same number of nodes), the number of segments owned by each node should be approximately the same.
> The current algorithm properly balances the primary owners of the segments, but the backup owners are not balanced, so a node can end up owning a lot more segments than expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2952) The REST endpoint should register with the modcluster subsystem
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-2952?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on ISPN-2952:
--------------------------------
Looks like the node *is* registered, but the context (/rest) is *not*
> The REST endpoint should register with the modcluster subsystem
> ---------------------------------------------------------------
>
> Key: ISPN-2952
> URL: https://issues.jboss.org/browse/ISPN-2952
> Project: Infinispan
> Issue Type: Feature Request
> Components: Server
> Affects Versions: 5.2.5.Final
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Fix For: 5.3.0.Final
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2952) The REST endpoint should register with the modcluster subsystem
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-2952?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-2952:
---------------------------------------
The context needs to be distributable.
> The REST endpoint should register with the modcluster subsystem
> ---------------------------------------------------------------
>
> Key: ISPN-2952
> URL: https://issues.jboss.org/browse/ISPN-2952
> Project: Infinispan
> Issue Type: Feature Request
> Components: Server
> Affects Versions: 5.2.5.Final
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Fix For: 5.3.0.Final
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0
[JBoss JIRA] (ISPN-2950) In distributed mode cache store data should be read through the main data owner (vs directly from the store)
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-2950?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero edited comment on ISPN-2950 at 3/20/13 11:59 AM:
-----------------------------------------------------------------
Why should the ClusterCacheLoader be deprecated? I think that's totally unrelated, I might still want to load data from a different grid instance, or another cache having a different configuration.
was (Author: sannegrinovero):
Why should the ClusterCacheLoader be deprecated? I think that's totally unrelated, I might still want to load data from a different grid instance.
> In distributed mode cache store data should be read through the main data owner (vs directly from the store)
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2950
> URL: https://issues.jboss.org/browse/ISPN-2950
> Project: Infinispan
> Issue Type: Feature Request
> Components: Loaders and Stores
> Reporter: Mircea Markus
> Assignee: Mircea Markus
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> Dist cache with a cache store(shared or not), k owned by \{N1, N2\}. k is read on N3. What currently happens at this stage, if k is not present in N3's memory (likely unless L1 is configured), the N3's cache store is queried and data is loaded from there. This has several drawback:
> - the data might already be in the memory of the owner node (N1,N2) so reading it from the disk is highly inefficient. Especially for hot data: data requested from various nodes at the same time (see also mailing list discussion around lucene query performance depending on this)
> - if this is a local cache store, it might contain stale data which would be returned to the user
> - for async configured cache store this would result in dirty reads, given that a change might be in the async store's memory but not in the store at the moment when it is in read by N3. (Note that using async stores still leaves place to inconsistencies when a node leaves, e.g. because of node crashing before managing to flush the async store.)
> This JIRA is about changing the distribution mode: when asked for a specific key, a node would only touch a cache store if it is an owner of that key, otherwise would first go to the main owner of the key to read the value from there. The ClusterCacheLoader should be deprecated as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
- 1
- 0