[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:28 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- we get the session corresponding to the session id from the ISPN cache (StatefulComponentInstanceInterceptor)
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:25 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:24 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code:xml}
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:23 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code:xml}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:19 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be?
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:15 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be?
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Need to check:
- when is the topology change received by node-1?
- what to do in case an owner node is no longer available?
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFCORE-952) Use WildFly Common for null param checks
by David Lloyd (JIRA)
[ https://issues.jboss.org/browse/WFCORE-952?page=com.atlassian.jira.plugin... ]
David Lloyd commented on WFCORE-952:
------------------------------------
This really represents about an hour or two of work at most, but it doesn't seem right to do such a broad (if shallow) change while so much else is going on. If you think it'd be fine though, I'll just whip up a PR tomorrow and close this one up.
> Use WildFly Common for null param checks
> ----------------------------------------
>
> Key: WFCORE-952
> URL: https://issues.jboss.org/browse/WFCORE-952
> Project: WildFly Core
> Issue Type: Task
> Reporter: David Lloyd
> Priority: Minor
>
> For each module, do the following:
> * Locate any/all null param check methods in the log msg
> * Replace them with calls to org.wildfly.common.Assert#checkNotNullParam or related method as needed
> * Replace the old null param check method with a comment that reserves the ID and shows that it was previously used for that purpose
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-3858) Infinispan cache configuration is not always applied to security-domain
by R O (JIRA)
[ https://issues.jboss.org/browse/WFLY-3858?page=com.atlassian.jira.plugin.... ]
R O commented on WFLY-3858:
---------------------------
I guess that problem is located somewhere on subsystems level. Sometimes security subsystem starts before infinispan subsystem is ready to use.
> Infinispan cache configuration is not always applied to security-domain
> -----------------------------------------------------------------------
>
> Key: WFLY-3858
> URL: https://issues.jboss.org/browse/WFLY-3858
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Affects Versions: 8.1.0.Final
> Reporter: Robert Tuck
> Assignee: Darran Lofthouse
> Attachments: debugger output.txt
>
>
> On Wildfly 8.1.0.Final, I have the following standalone-ha.xml:
> <subsystem xmlns="urn:jboss:domain:infinispan:2.0">
> ...
> <cache-container name="security" default-cache="auth-cache" start="EAGER">
> <transport cluster="${cluster.name}_SEC" lock-timeout="60000"/>
> <replicated-cache name="auth-cache" batching="true" mode="ASYNC">
> <eviction strategy="LRU" max-entries="10000"/>
> <expiration lifespan="60000"/>
> </replicated-cache>
> </cache-container>
> </subsystem>
> ...
> <subsystem xmlns="urn:jboss:domain:security:1.2">
> <security-domains>
> ...
> <security-domain name="OAuth-Consumer" cache-type="infinispan">
> <authentication>
> <login-module code="com.idbs.ewb.server.auth.module.OAuthConsumerLoginModule" flag="sufficient"
> module="deployment.ewb-server-ear.ear">
> <module-option name="allowedConsumerAuthFailures" value="-1"/>
> <module-option name="consumerLoginFailureTimeoutMs" value="3000"/>
> </login-module>
> </authentication>
> </security-domain>
> </security-domains>
> </subsystem>
> After startup the OAuth-Consumer security domain cache "auth-cache" is not always configured with the specified settings (~50% of the time). This can be verified by monitoring the jboss.infinispan nodes with JConsole and retrieving the cache settings, and tracking the cache hits during logins. This shows that succesful logins are cached but do not expire after the expected 60s, and that the expiration lifespan is set to -1 rather than 60000, as are eviction max entries.
> After some debugging I have narrowed down the problem to some code that gets called from inside org.jboss.as.security.plugins.JNDIBasedSecurityManagement:
> public SecurityDomainContext createSecurityDomainContext(String securityDomain, Object cacheFactory) throws Exception {
> log.debugf("Creating SDC for domain=" + securityDomain);
> AuthenticationManager am = createAuthenticationManager(securityDomain);
> // create authentication cache
> if (cacheFactory instanceof EmbeddedCacheManager) {
> EmbeddedCacheManager cacheManager = EmbeddedCacheManager.class.cast(cacheFactory);
> @SuppressWarnings("rawtypes")
> Cache cache = null;
> if (cacheManager != null) {
> // TODO override global settings with security domain specific
> ConfigurationBuilder builder = new ConfigurationBuilder();
> Configuration baseCfg = cacheManager.getCacheConfiguration("auth-cache");
> ^^^^ This call here doesn’t always return the correct configuration, baseCfg is then null.
> if (baseCfg != null) {
> builder.read(baseCfg);
> }
> cacheManager.defineConfiguration(securityDomain, builder.build());
> cache = cacheManager.getCache(securityDomain);
> }
> if (cache != null && am instanceof CacheableManager) {
> @SuppressWarnings({ "unchecked", "rawtypes" })
> CacheableManager<Map, Principal> cm = (CacheableManager<Map, Principal>) am;
> cm.setCache(cache);
> }
> } else if (cacheFactory instanceof DefaultAuthenticationCacheFactory) {
> <…>
> }
> getCacheConfiguration(String) is implemented inside org.infinispan.manager.DefaultCacheManager:
> @Override
> public Configuration getCacheConfiguration(String name) {
> Configuration configuration = configurationOverrides.get(name);
> if (configuration == null && cacheExists(name)) {
> return defaultConfiguration;
> }
> return configuration;
> }
> Seems like the condition configuration == null occurs when the cache doesn’t exist, therefore it returns null. This appears to be a race condition between this code and where it gets registered in wireAndStartCache(String).
>
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 3:53 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Need to check:
- when is the topology change received by node-1?
- what to do in case an owner node is no longer available?
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation and returns an exception to the client
Need to check:
- when is the topology change received by node-1?
- what to do in case an owner node is no longer available?
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 3:52 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation and returns an exception to the client
Need to check:
- when is the topology change received by node-1?
- what to do in case an owner node is no longer available?
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for invocation Y
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation and returns an exception to the client
Need to check:
- when is the topology change received by node-1?
- what to do in case an owner node is no longer available?
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
- 1
- 0