[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
Paul Ferraro commented on WFLY-10736:
-------------------------------------
The workaround should read: "Add delay between server kill and restart" This gives cluster the opportunity to detect and respond to the dropped member before adding a new member with the same physical address.
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Fix For: 14.0.0.CR1
>
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, Lodh2TestCase.testRemoteJcaInboundOnly2.zip, clusterKilTest.zip, logs-traces.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-9407) Upgrade Artemis 2.6.3.jbossorg-001
by ehsavoie Hugonnet (JIRA)
[ https://issues.jboss.org/browse/WFLY-9407?page=com.atlassian.jira.plugin.... ]
ehsavoie Hugonnet updated WFLY-9407:
------------------------------------
Summary: Upgrade Artemis 2.6.3.jbossorg-001 (was: Upgrade Artemis 2.6.0)
> Upgrade Artemis 2.6.3.jbossorg-001
> ----------------------------------
>
> Key: WFLY-9407
> URL: https://issues.jboss.org/browse/WFLY-9407
> Project: WildFly
> Issue Type: Component Upgrade
> Components: JMS
> Reporter: Jeff Mesnil
> Assignee: Jeff Mesnil
> Labels: downstream_dependency
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
Miroslav Novak updated WFLY-10736:
----------------------------------
Workaround Description:
-Configure non-blocking state transfer via: <state-transfer timeout="0"/>-
*Note:*
Workaround does not fix case when MDB is deployed on restarted server. Infinispan subsystem fails to start and MDB does not activate. Currently there is no workaround.
was:
Configure non-blocking state transfer via: <state-transfer timeout="0"/>
*Note:*
Workaround does not fix case when MDB is deployed on restarted server. Infinispan subsystem fails to start and MDB does not activate.
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Fix For: 14.0.0.CR1
>
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, Lodh2TestCase.testRemoteJcaInboundOnly2.zip, clusterKilTest.zip, logs-traces.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-10464) ISPN000482: Cannot create remote transaction X, already completed in ASYM_ENCRYPT scenario (following "received message without encrypt header from perf21; dropping it")
by Gunther v. Wolffersdorff (JIRA)
[ https://issues.jboss.org/browse/WFLY-10464?page=com.atlassian.jira.plugin... ]
Gunther v. Wolffersdorff commented on WFLY-10464:
-------------------------------------------------
Same problem.
Randomly (1x per hour in production) we get on 3 of 4 servers:
{code}
07:52:35,039 UTC WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p13-t395) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[InvalidateCommand{keys=[de.alvara.model.entity.ServiceContract#7025094, de.alvara.model.entity.ServiceContract#7025057, de.alvara.model.entity.ServiceContract#7025090, de.alvara.model.entity.ServiceContract#7025059, de.alvara.model.entity.ServiceContract#7025098, de.alvara.model.entity.ServiceContract#7025067, de.alvara.model.entity.ServiceContract#7025079, de.alvara.model.entity.ServiceContract#7027505...<26 other elements>]}], onePhaseCommit=true, retried=false, gtx=GlobalTx:app03:server-2:831722, cacheName='Icc-ear-2018.3-24414.ear#icc-pu.de.alvara.model.entity.ServiceContract', topologyId=17}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
at java.base/java.lang.Thread.run(Unknown Source)
{code}
and on 1 server we get:
{code}
07:52:35,043 UTC ERROR [org.infinispan.hibernate.cache.commons.access.TxInvalidationInterceptor] (thread-1652,ejb,app03:server-2) HHH025020: Unable to broadcast invalidations as a part of the prepare phase. Rolling back.: java.util.concurrent.CompletionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.AbstractRequest.completeExceptionally(AbstractRequest.java:74)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
... 29 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
07:52:35,043 UTC ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-1652,ejb,app03:server-2) ISPN000136: Error executing command PrepareCommand, writing keys [de.alvara.model.entity.ServiceContract#7025094, de.alvara.model.entity.ServiceContract#7025057, de.alvara.model.entity.ServiceContract#7025090, de.alvara.model.entity.ServiceContract#7025059, de.alvara.model.entity.ServiceContract#7025098, de.alvara.model.entity.ServiceContract#7025067, de.alvara.model.entity.ServiceContract#7025079, de.alvara.model.entity.ServiceContract#7027505...<26 other elements>]: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
07:52:35,044 UTC ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (Batch Thread - 15 | !{f879f2f2-0080-4ea0-b226-72689bd266d2}! > orderbookCalculatorJob > exec) ISPN000255: Error while processing prepare: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:119)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions@13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core@3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common@13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private@1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet@13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller@5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace]
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace]
[CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace]
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
[CIRCULAR REFERENCE:org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed]
07:52:35,045 UTC WARN [com.arjuna.ats.arjuna] (Batch Thread - 15 | !{f879f2f2-0080-4ea0-b226-72689bd266d2}! > orderbookCalculatorJob > exec) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffffc0a82a17:10fdf8f5:5b81c4c6:2eab36, org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@1f4a3227 >: org.infinispan.commons.CacheException: Could not prepare.
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:839)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions@13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core@3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common@13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private@1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet@13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller@5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: javax.transaction.xa.XAException
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:139)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
... 82 more
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:119)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions@13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core@3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common@13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private@1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet@13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller@5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace]
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
07:52:35,049 UTC ERROR [org.jboss.as.ejb3.invocation] (Batch Thread - 15 | !{f879f2f2-0080-4ea0-b226-72689bd266d2}! > orderbookCalculatorJob > exec) WFLYEJB0034: EJB Invocation failed on component OrderbookCalculatorServiceImpl for method public abstract de.alvara.api.batch.ExitStatus de.alvara.api.scheduler.JobableService.runInBatch(java.util.Properties) throws de.alvara.api.scheduler.JobException: javax.ejb.EJBTransactionRolledbackException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:114)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core@3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common@13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private@1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet@13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller@5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
... 67 more
Suppressed: javax.transaction.RollbackException: WFTXN0061: Transaction is marked rollback-only
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.setRollbackOnly(AbstractTransaction.java:96)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.setRollbackOnly(LocalTransaction.java:149)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.markTxForRollback(InvocationContextInterceptor.java:172)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.rethrowException(InvocationContextInterceptor.java:131)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.lambda$new$0(InvocationContextInterceptor.java:62)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.InvocationExceptionFunction.apply(InvocationExceptionFunction.java:21)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:118)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:81)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:30)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.AbstractRequest.completeExceptionally(AbstractRequest.java:74)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.infinispan.commons.CacheException: Could not prepare.
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:839)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions@13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
... 72 more
Caused by: javax.transaction.xa.XAException
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:139)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
... 82 more
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:119)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions@13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core@3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common@13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private@1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet@13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller@5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace]
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
07:52:35,064 UTC WARN [org.jberet] (Batch Thread - 15 | !{f879f2f2-0080-4ea0-b226-72689bd266d2}! > orderbookCalculatorJob > exec) JBERET000001: Failed to run batchlet org.jberet.job.model.RefArtifact@2777a5d1: javax.ejb.EJBTransactionRolledbackException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:114)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core@3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common@13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private@1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet@13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller@5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
... 67 more
Suppressed: javax.transaction.RollbackException: WFTXN0061: Transaction is marked rollback-only
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.setRollbackOnly(AbstractTransaction.java:96)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.setRollbackOnly(LocalTransaction.java:149)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.markTxForRollback(InvocationContextInterceptor.java:172)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.rethrowException(InvocationContextInterceptor.java:131)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.lambda$new$0(InvocationContextInterceptor.java:62)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.InvocationExceptionFunction.apply(InvocationExceptionFunction.java:21)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:118)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:81)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:30)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.AbstractRequest.completeExceptionally(AbstractRequest.java:74)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.infinispan.commons.CacheException: Could not prepare.
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:839)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions@13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
... 72 more
Caused by: javax.transaction.xa.XAException
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:139)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
... 82 more
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:46)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:18)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356)
at org.jgroups@4.0.11.Final//org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups@4.0.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:340)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FORK.up(FORK.java:134)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:267)
at org.jgroups@4.0.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
at org.jgroups@4.0.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.jboss.as.clustering.jgroups@13.0.0.Final//org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-2, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app04:server-1, see cause for remote stack trace
... 34 more
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
Suppressed: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan@9.2.4.Final//org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:119)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:102)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.beforeCompletion(TransactionTable.java:837)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:28)
at org.jboss.as.transactions@13.0.0.Final//org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.beforeCompletion(JCAOrderedLastSynchronizationList.java:113)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at org.wildfly.transaction.client@1.1.2.Final//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:146)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.weld.core@3.0.4.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at org.jboss.as.weld.common@13.0.0.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.elytron-private@1.3.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee@13.0.0.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3@13.0.0.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:100)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
at org.jboss.ejb-client@4.0.10.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
at deployment.SchedulerEar-2018.3-24414.ear//com.sun.proxy.$Proxy360.runInBatch(Unknown Source)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.JobableServiceBatchlet.run(JobableServiceBatchlet.java:34)
at deployment.SchedulerEar-2018.3-24414.ear.Scheduler.jar//de.alvara.batch.batchlet.AbstractSimpleBatchlet.process(AbstractSimpleBatchlet.java:58)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:72)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:229)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60)
at org.wildfly.extension.batch.jberet@13.0.0.Final//org.wildfly.extension.batch.jberet.deployment.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:184)
at org.wildfly.extension.request-controller@5.0.0.Final//org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:494)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$3.run(JobExecutor.java:161)
at org.jberet.jberet-core@1.2.4.Final//org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
[CIRCULAR REFERENCE:org.infinispan.remoting.RemoteException: ISPN000217: Received exception from app03:server-1, see cause for remote stack trace]
Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:app03:server-2:831722, already completed
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
at org.infinispan@9.2.4.Final//org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
at org.infinispan@9.2.4.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan@9.2.4.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.wildfly.clustering.service@13.0.0.Final//org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
... 1 more
{code}
> ISPN000482: Cannot create remote transaction X, already completed in ASYM_ENCRYPT scenario (following "received message without encrypt header from perf21; dropping it")
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10464
> URL: https://issues.jboss.org/browse/WFLY-10464
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Critical
>
> Affected scenario: eap-7x-failover-http-session-shutdown-dist-sync-auth-asymEncrypt (scenario with "AUTH" and "ASYM_ENCRYPT" protocols enabled).
> In server logs, we can see lots of WARN/ERROR messages logged.
> With 2000 clients, none of them is bound to a specific time/event (i.e. failover).
> When lowering number of clients to 10, Excpetion #1 starts occurring right after each failover, #2 + #3 occur after node rejoins the cluster.
> {code:title=#1 ISPN000482: Cannot create remote transaction X, already completed}
> [JBossINF] [0m[33m04:44:50,005 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p5-t3) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(r6mU8aS9wDfsmP7rAJ1892KSrjgnmzoTRytiyrNt)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf21:49128}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf21:49128, already completed
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:375)
> [JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:368)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:348)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> {code}
> {code:title=#2 ISPN000476: Timed out waiting for responses for request 47698 from X}
> [JBossINF] [0m[31m04:46:24,448 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p12-t1) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47698 from perf21
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> {code:title=#3 CIRCULAR REFERENCE -> UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from X}
> [JBossINF] [0m[31m04:46:24,053 ERROR [io.undertow.request] (default task-98) UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from perf21
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:485)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:528)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:658)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:200)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:176)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412)
> [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241)
> [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from perf21
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from perf21
> [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:485)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:528)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:658)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:200)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:176)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412)
> [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241)
> [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] ... 1 more
> [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 47460 from perf21]
> {code}
> {code:title=#4 received message without encrypt header}
> [JBossINF] [0m[31m04:45:10,048 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-9,ejb,perf18) perf18: received message without encrypt header from perf21; dropping it
> {code}
> Client gets valid responses till approx. 30 seconds after first failover, then Read timeouts start occuring. As far as I can tell, no valid response is received from this point.
> Link to server log - 2000 clients:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
> Link to client log - 2000 clients:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
> Link to server log - 10 clients:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
> Link to client log - 10 clients:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
Miroslav Novak updated WFLY-10736:
----------------------------------
Workaround: (was: Workaround Exists)
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Fix For: 14.0.0.CR1
>
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, Lodh2TestCase.testRemoteJcaInboundOnly2.zip, clusterKilTest.zip, logs-traces.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (DROOLS-2706) New "Scenario Test" asset list option (UX)
by Ivo Bek (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2706?page=com.atlassian.jira.plugi... ]
Ivo Bek resolved DROOLS-2706.
-----------------------------
Resolution: Done
> New "Scenario Test" asset list option (UX)
> ------------------------------------------
>
> Key: DROOLS-2706
> URL: https://issues.jboss.org/browse/DROOLS-2706
> Project: Drools
> Issue Type: Task
> Components: Scenario Simulation and Testing
> Reporter: Liz Clayton
> Assignee: Ivo Bek
> Labels: ScenarioSimulation, UX, UXTeam
>
> As a practitioner - I want to be able to distinguish the new Scenario Test tool option, from the existing one, within the Asset list so that I can quickly/easily create a test using the new editor.
> *Verification conditions:*
> The new Scenario Test tool is distinguishable from the existing test tool within the asset list, through visual inspection by the user (no hover/click required.)
> Notes:
> Need a way to identify "experimental" tools from existing (to be deprecated?) tools. Solution might include distinct names for each, or parenthetical append (e.g. "experimental, beta, preview...) Or visual indicator, such as an icon, flag, etc.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (SWSQE-392) Fix Jenkins Job To Run Suite From Kiali Repo
by Matt Mahoney (JIRA)
Matt Mahoney created SWSQE-392:
----------------------------------
Summary: Fix Jenkins Job To Run Suite From Kiali Repo
Key: SWSQE-392
URL: https://issues.jboss.org/browse/SWSQE-392
Project: Kiali QE
Issue Type: QE Task
Reporter: Matt Mahoney
Assignee: Matt Mahoney
kiali-qe-rest suite has now moved into Kiali repo tests/e2e.
Adjust Jenkins job to now run suite from Kiali repo
Ref: [https://github.com/kiali/kiali/tree/master/tests/e2e|https://github.com/k...]
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (DROOLS-2874) Select a data type, inline, from DMN decision tables.
by Liz Clayton (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2874?page=com.atlassian.jira.plugi... ]
Liz Clayton commented on DROOLS-2874:
-------------------------------------
[~manstis] design is done, waiting for review/approval.
> Select a data type, inline, from DMN decision tables.
> -----------------------------------------------------
>
> Key: DROOLS-2874
> URL: https://issues.jboss.org/browse/DROOLS-2874
> Project: Drools
> Issue Type: Story
> Components: DMN Editor
> Reporter: Liz Clayton
> Assignee: Liz Clayton
> Labels: UX, UXTeam, drools-tools
> Attachments: DataType selection ('Properties Panel' and 'in-grid').png, Screen Shot 2018-08-10 at 10.23.36 AM.png, Screen Shot 2018-08-24 at 8.38.37 AM.png, pop-overc.png, select.png
>
>
> *Background*
> Persona: Business analyst or Rules practitioner
> Use Cases:
> * From the DMN canvas view - as a user I want to *select* a data type inline from a decision table.
> *
> Functional considerations/ pre conditions:
> * Actions will be offered or spawned inline within the cells of decision tables.
> * Design needs to be consistent with Stunner and PF components.
> Verification conditions:
> * Scrum team and PO review.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (DROOLS-2522) [DMN Designer] Palette aesthetics
by Liz Clayton (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2522?page=com.atlassian.jira.plugi... ]
Liz Clayton updated DROOLS-2522:
--------------------------------
Labels: (was: UX UXTeam)
> [DMN Designer] Palette aesthetics
> ---------------------------------
>
> Key: DROOLS-2522
> URL: https://issues.jboss.org/browse/DROOLS-2522
> Project: Drools
> Issue Type: Bug
> Components: DMN Editor
> Affects Versions: 7.8.0.Final
> Reporter: Jozef Marko
> Assignee: Michael Anstis
> Priority: Minor
> Labels: drools-tools
> Attachments: palette.png
>
>
> The palette css styling has changed. There appeared margins between items in palette, what is probably not problem, however even when user click to this margin (to space between two item in palette) still some palette item is selected.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0
[JBoss JIRA] (DROOLS-2522) [DMN Designer] Palette aesthetics
by Liz Clayton (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2522?page=com.atlassian.jira.plugi... ]
Liz Clayton updated DROOLS-2522:
--------------------------------
Labels: drools-tools (was: )
> [DMN Designer] Palette aesthetics
> ---------------------------------
>
> Key: DROOLS-2522
> URL: https://issues.jboss.org/browse/DROOLS-2522
> Project: Drools
> Issue Type: Bug
> Components: DMN Editor
> Affects Versions: 7.8.0.Final
> Reporter: Jozef Marko
> Assignee: Michael Anstis
> Priority: Minor
> Labels: drools-tools
> Attachments: palette.png
>
>
> The palette css styling has changed. There appeared margins between items in palette, what is probably not problem, however even when user click to this margin (to space between two item in palette) still some palette item is selected.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 8 months
- 1
- 0