[
https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin....
]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:24 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on
both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is
started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for
the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction
(CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session
id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from
its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this
issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to
the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15
seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few
pending invocations at the time of shutdown which the suspend mechanism needs to wait for
to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only
after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception
arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the
latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as
quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner
node is no longer available in the current topology (as is the case here with node-0 and
the session instance it is holding) despite the fact that the current topology says it
should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache
entry and return that as a result of the invocation, rather than throw an exception if the
owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb"
default-cache="dist"
module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC"
l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on
both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is
started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for
the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction
(CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session
id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from
its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this
issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to
the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15
seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few
pending invocations at the time of shutdown which the suspend mechanism needs to wait for
to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only
after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception
arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the
latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as
quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner
node is no longer available in the current topology (as is the case here with node-0 and
the session instance it is holding) despite the fact that the current topology says it
should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache
entry and return that as a result of the invocation, rather than throw an exception if the
owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb"
default-cache="dist"
module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC"
l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code:xml}
RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails
with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is
an invocation not belonging to an on-going transaction, so it does not accept new
invocations."
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key: WFLY-7029
URL:
https://issues.jboss.org/browse/WFLY-7029
Project: WildFly
Issue Type: Bug
Components: Clustering
Affects Versions: 10.1.0.Final
Reporter: Radoslav Husar
Assignee: Richard Achmatowicz
https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv...
{noformat}
04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping
container=container-0, timeout=15
04:31:43,324 INFO
[org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main)
Manual stopping of a server instance with timeout=15
&#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4)
WFLYSRV0211: Suspending server with 15000 ms timeout.
&#27;[0m04:31:43,343 INFO
[org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting
"config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on
node node-0 with exception:: javax.ejb.NoSuchEJBException: No such
EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
at
org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
at
org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
at
org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
at
org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
&#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown)
WFLYSRV0241: Shutting down in response to management operation 'shutdown'
&#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC
service thread 1-2) WFLYUT0019: Host default-host stopping
&#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService
Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
&#27;[0m&#27;[0m04:31:43,445 INFO
[org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74)
WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
&#27;[0m&#27;[0m04:31:43,459 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2)
ISPN000080: Disconnecting JGroups channel hibernate
&#27;[0m&#27;[0m04:31:43,459 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2)
ISPN000082: Stopping the RpcDispatcher for channel hibernate
&#27;[0m&#27;[0m04:31:43,463 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5)
ISPN000080: Disconnecting JGroups channel server
&#27;[0m&#27;[0m04:31:43,464 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5)
ISPN000082: Stopping the RpcDispatcher for channel server
&#27;[0m&#27;[0m04:31:43,471 INFO
[org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29)
ISPN000029: Passivating all entries to disk
&#27;[0m&#27;[0m04:31:43,472 INFO
[org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29)
ISPN000030: Passivated 2 entries in 0 milliseconds
&#27;[0m&#27;[0m04:31:43,501 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4)
ISPN000080: Disconnecting JGroups channel web
&#27;[0m&#27;[0m04:31:43,501 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4)
ISPN000082: Stopping the RpcDispatcher for channel web
&#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan]
(ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache
from ejb container
&#27;[0m&#27;[0m04:31:43,505 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1)
ISPN000080: Disconnecting JGroups channel testDbPersistence
&#27;[0m&#27;[0m04:31:43,505 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1)
ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
&#27;[0m&#27;[33m04:31:43,502 WARN
[org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler]
(thread-11,ee,node-0) ISPN000071: Caught exception when handling command
LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID
[6951504948515351686656515249505365655750686851525048706670486766]],
flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}:
org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in
'STOPPING' state and this is an invocation not belonging to an on-going
transaction, so it does not accept new invocations. Either restart it or recreate the
cache container.
at
org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
at
org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
at
org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
at
org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
at
org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
at
org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
at
org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
at
org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
at
org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
at
org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
at
org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
at
org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
at
org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
at
org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
at
org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
at
org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
at org.jgroups.JChannel.up(JChannel.java:738)
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
at org.jgroups.stack.Protocol.up(Protocol.java:380)
at org.jgroups.protocols.FORK.up(FORK.java:114)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
at org.jgroups.protocols.FD.up(FD.java:260)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
at org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups.protocols.MPING.up(MPING.java:178)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.lang.Thread.run(Thread.java:745)
&#27;[0m&#27;[0m04:31:43,517 INFO
[org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006:
Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
&#27;[0m&#27;[0m04:31:43,517 INFO
[org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010:
Unbound data source [java:jboss/datasources/ExampleDS]
&#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC
service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
&#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc]
(MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
&#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan]
(ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb
container
&#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting
"config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID
b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint
"config-based-ejb-client-endpoint" <1c33eee> can no longer process
messages
&#27;[0m04:31:43,576 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3)
ISPN000080: Disconnecting JGroups channel ejb
&#27;[0m&#27;[0m04:31:43,576 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3)
ISPN000082: Stopping the RpcDispatcher for channel ejb
&#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra]
(ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
&#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC
service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
&#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC
service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to
[::1]:8009
&#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC
service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
&#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC
service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to
[::1]:8443
&#27;[0m&#27;[31m04:31:43,585 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136:
Error executing command GetKeyValueCommand, writing keys []:
org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see
cause for remote stack trace
at
org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at
org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
at org.jgroups.JChannel.up(JChannel.java:738)
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
at org.jgroups.stack.Protocol.up(Protocol.java:380)
at org.jgroups.protocols.FORK.up(FORK.java:114)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
at org.jgroups.protocols.FD.up(FD.java:260)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
at org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups.protocols.MPING.up(MPING.java:178)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.commons.CacheException: Problems invoking command.
at
org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
at
org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
at
org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
at
org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
... 27 more
Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324:
remote-failover-test.jar is in 'STOPPING' state and this is an invocation not
belonging to an on-going transaction, so it does not accept new invocations. Either
restart it or recreate the cache container.
at
org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
at
org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
at
org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
at
org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
at
org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
at
org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
at
org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
at
org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
at
org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
at
org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
at
org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
at
org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
at
org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
at
org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
... 33 more
&#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default
task-11) WFLYEJB0034: EJB Invocation failed on component
SlowToDestroyStatefulIncrementorBean for method public abstract
org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result
org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment():
javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received
exception from node-0, see cause for remote stack trace
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
at
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at
org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
at
org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
at
org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
at
org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
at
org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
at
org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
at
org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
at
org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from
node-0, see cause for remote stack trace
at
org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at
org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
at org.jgroups.JChannel.up(JChannel.java:738)
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
at org.jgroups.stack.Protocol.up(Protocol.java:380)
at org.jgroups.protocols.FORK.up(FORK.java:114)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
at org.jgroups.protocols.FD.up(FD.java:260)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
at org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups.protocols.MPING.up(MPING.java:178)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
... 1 more
Caused by: org.infinispan.commons.CacheException: Problems invoking command.
at
org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
at
org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
at
org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
at
org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
... 27 more
Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324:
remote-failover-test.jar is in 'STOPPING' state and this is an invocation not
belonging to an on-going transaction, so it does not accept new invocations. Either
restart it or recreate the cache container.
at
org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
at
org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
at
org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
at
org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
at
org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
at
org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
at
org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
at
org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
at
org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
at
org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
at
org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
at
org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
at
org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
at
org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
... 33 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)