[jboss-jira] [JBoss JIRA] (WFLY-10464) ISPN000482: Cannot create remote transaction X, already completed in ASYM_ENCRYPT scenario (following "received message without encrypt header from perf21; dropping it")
Gunther v. Wolffersdorff (JIRA)
issues at jboss.org
Tue Aug 28 10:15:04 EDT 2018
[ https://issues.jboss.org/browse/WFLY-10464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13625174#comment-13625174 ]
Gunther v. Wolffersdorff commented on WFLY-10464:
-------------------------------------------------
Same problem.
Randomly (1x per hour in production) we get on 3 of 4 servers:
{code}
07:52:35,039 UTC WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p13-t395) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[InvalidateCommand{keys=[de.alvara.model.entity.ServiceContract#7025094, de.alvara.model.entity.ServiceContract#7025057, de.alvara.model.entity.ServiceContract#7025090, de.alvara.model.entity.ServiceContract#7025059, de.alvara.model.entity.ServiceContract#7025098, de.alvara.model.entity.ServiceContract#7025067, de.alvara.model.entity.ServiceContract#7025079, de.alvara.model.entity.ServiceContract#7027505...<26 other elements>]}], onePhaseCommit=true, retried=false, gtx=GlobalTx:app03:server-2:831722, cacheName='Icc-ear-2018.3-24414.ear#icc-pu.de.alvara.model.entity.ServiceContract', topologyId=17}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
at java.base/java.lang.Thread.run(Unknown Source)
{code}
and on 1 server we get:
{code}
07:52:35,043 UTC ERROR [org.infinispan.hibernate.cache.commons.access.TxInvalidationInterceptor] (thread-1652,ejb,app03:server-2) HHH025020: Unable to broadcast invalidations as a part of the prepare phase. Rolling back.: java.util.concurrent.CompletionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.AbstractRequest.completeExceptionally(AbstractRequest.java:74)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
... 29 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
07:52:35,043 UTC ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-1652,ejb,app03:server-2) ISPN000136: Error executing command PrepareCommand, writing keys [de.alvara.model.entity.ServiceContract#7025094, de.alvara.model.entity.ServiceContract#7025057, de.alvara.model.entity.ServiceContract#7025090, de.alvara.model.entity.ServiceContract#7025059, de.alvara.model.entity.ServiceContract#7025098, de.alvara.model.entity.ServiceContract#7025067, de.alvara.model.entity.ServiceContract#7025079, de.alvara.model.entity.ServiceContract#7027505...<26 other elements>]: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
07:52:35,044 UTC ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (Batch Thread - 15 | !{f879f2f2-0080-4ea0-b226-72689bd266d2}! > orderbookCalculatorJob > exec) ISPN000255: Error while processing prepare: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:119)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions at 13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core at 3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common at 13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private at 1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common at 1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet at 13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller at 5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads at 2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace]
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace]
[CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace]
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
[CIRCULAR REFERENCE:org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed]
07:52:35,045 UTC WARN [com.arjuna.ats.arjuna] (Batch Thread - 15 | !{f879f2f2-0080-4ea0-b226-72689bd266d2}! > orderbookCalculatorJob > exec) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffffc0a82a17:10fdf8f5:5b81c4c6:2eab36, org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization at 1f4a3227 >: org.infinispan.commons.CacheException: Could not prepare.
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:839)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions at 13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core at 3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common at 13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private at 1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common at 1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet at 13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller at 5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads at 2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: javax.transaction.xa.XAException
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:139)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
... 82 more
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:119)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions at 13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core at 3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common at 13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private at 1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common at 1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet at 13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller at 5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads at 2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace]
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
07:52:35,049 UTC ERROR [org.jboss.as.ejb3.invocation] (Batch Thread - 15 | !{f879f2f2-0080-4ea0-b226-72689bd266d2}! > orderbookCalculatorJob > exec) WFLYEJB0034: EJB Invocation failed on component OrderbookCalculatorServiceImpl for method public abstract de.alvara.api.batch.ExitStatus de.alvara.api.scheduler.JobableService.runInBatch(java.util.Properties) throws de.alvara.api.scheduler.JobException: javax.ejb.EJBTransactionRolledbackException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:114)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core at 3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common at 13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private at 1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common at 1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet at 13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller at 5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads at 2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
... 67 more
Suppressed: javax.transaction.RollbackException: WFTXN0061: Transaction is marked rollback-only
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.setRollbackOnly(AbstractTransaction.java:96)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.setRollbackOnly(LocalTransaction.java:149)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.markTxForRollback(InvocationContextInterceptor.java:172)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.rethrowException(InvocationContextInterceptor.java:131)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.lambda$new$0(InvocationContextInterceptor.java:62)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.InvocationExceptionFunction.apply(InvocationExceptionFunction.java:21)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:118)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:81)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:30)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.AbstractRequest.completeExceptionally(AbstractRequest.java:74)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.infinispan.commons.CacheException: Could not prepare.
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:839)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions at 13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
... 72 more
Caused by: javax.transaction.xa.XAException
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:139)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
... 82 more
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:119)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions at 13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core at 3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common at 13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private at 1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common at 1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet at 13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller at 5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads at 2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace]
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
07:52:35,064 UTC WARN [org.jberet] (Batch Thread - 15 | !{f879f2f2-0080-4ea0-b226-72689bd266d2}! > orderbookCalculatorJob > exec) JBERET000001: Failed to run batchlet org.jberet.job.model.RefArtifact at 2777a5d1: javax.ejb.EJBTransactionRolledbackException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:114)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core at 3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common at 13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private at 1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common at 1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet at 13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller at 5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads at 2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
... 67 more
Suppressed: javax.transaction.RollbackException: WFTXN0061: Transaction is marked rollback-only
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.setRollbackOnly(AbstractTransaction.java:96)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.setRollbackOnly(LocalTransaction.java:149)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.markTxForRollback(InvocationContextInterceptor.java:172)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.rethrowException(InvocationContextInterceptor.java:131)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.lambda$new$0(InvocationContextInterceptor.java:62)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.InvocationExceptionFunction.apply(InvocationExceptionFunction.java:21)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:118)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:81)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:30)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.AbstractRequest.completeExceptionally(AbstractRequest.java:74)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.infinispan.commons.CacheException: Could not prepare.
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:839)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions at 13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
... 72 more
Caused by: javax.transaction.xa.XAException
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:139)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
... 82 more
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups at 4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups at 4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups at 4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups at 4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups at 13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan at 9.2.4.Final//org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:119)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions at 13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client at 1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core at 3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common at 13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private at 1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation at 1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee at 13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3 at 13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common at 1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client at 4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet at 13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller at 5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core at 1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads at 2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace]
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan at 9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan at 9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan at 9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service at 13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
{code}
> ISPN000482: Cannot create remote transaction X, already completed in ASYM_ENCRYPT scenario (following "received message without encrypt header from perf21; dropping it")
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10464
> URL: https://issues.jboss.org/browse/WFLY-10464
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Critical
>
> Affected scenario: eap-7x-failover-http-session-shutdown-dist-sync-auth-asymEncrypt (scenario with "AUTH" and "ASYM_ENCRYPT" protocols enabled).
> In server logs, we can see lots of WARN/ERROR messages logged.
> With 2000 clients, none of them is bound to a specific time/event (i.e. failover).
> When lowering number of clients to 10, Excpetion #1 starts occurring right after each failover, #2 + #3 occur after node rejoins the cluster.
> {code:title=#1 ISPN000482: Cannot create remote transaction X, already completed}
> [JBossINF] [0m[33m04:44:50,005 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p5-t3) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(r6mU8aS9wDfsmP7rAJ1892KSrjgnmzoTRytiyrNt)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf21:49128}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf21:49128, already completed
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
> [JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> {code}
> {code:title=#2 ISPN000476: Timed out waiting for responses for request 47698 from X}
> [JBossINF] [0m[31m04:46:24,448 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p12-t1) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47698 from perf21
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> {code:title=#3 CIRCULAR REFERENCE -> UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from X}
> [JBossINF] [0m[31m04:46:24,053 ERROR [io.undertow.request] (default task-98) UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from perf21
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:485)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:528)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:658)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:200)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:176)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412)
> [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241)
> [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from perf21
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from perf21
> [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:485)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:528)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:658)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:200)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:176)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412)
> [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241)
> [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] ... 1 more
> [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from perf21]
> {code}
> {code:title=#4 received message without encrypt header}
> [JBossINF] [0m[31m04:45:10,048 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-9,ejb,perf18) perf18: received message without encrypt header from perf21; dropping it
> {code}
> Client gets valid responses till approx. 30 seconds after first failover, then Read timeouts start occuring. As far as I can tell, no valid response is received from this point.
> Link to server log - 2000 clients:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-sync-auth-asymEncrypt/28/console-perf18/
> Link to client log - 2000 clients:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-sync-auth-asymEncrypt/28/console-perf17/
> Link to server log - 10 clients:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-sync-auth-asymEncrypt/29/console-perf20/
> Link to client log - 10 clients:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-sync-auth-asymEncrypt/29/console-perf17/
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
More information about the jboss-jira
mailing list