[JBoss JIRA] (JGRP-2098) Discovery: reduce messages when IpAddressUUID is used
by Neal Dillman (JIRA)
[ https://issues.jboss.org/browse/JGRP-2098?page=com.atlassian.jira.plugin.... ]
Neal Dillman commented on JGRP-2098:
------------------------------------
I believe that discovery should be limited to finding the coordinator. IP address mapping, logical names, what have you, can all be put into a separate, optional protocol.
Further, I am not convinced that discovery is even really needed in the manner in which it is currently done. For example, if one were to listen for sendInfo() messages from MERGE3, one would be able to determine who the coordinator is and act upon that. In other words, I believe we may wish to handle discovery as a byproduct of another protocol.
> Discovery: reduce messages when IpAddressUUID is used
> -----------------------------------------------------
>
> Key: JGRP-2098
> URL: https://issues.jboss.org/browse/JGRP-2098
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0
>
>
> Since IpAddressUUID already contains the physical address, we don't need to exchange physical addresses in the discovery phase.
> Investigate whether this leads to reduced messaging in discovery, ie. only the coords might send a response. Once the new member has the view, it automatically knows the IP addresses and ports of all members, as the addresses in the view are of type IpAddressUUID.
> Also investigate whether address:logical_name associations should be done in a separate protocol, e.g. NAMING.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (JGRP-2098) Discovery: reduce messages when IpAddressUUID is used
by Neal Dillman (JIRA)
[ https://issues.jboss.org/browse/JGRP-2098?page=com.atlassian.jira.plugin.... ]
Neal Dillman edited comment on JGRP-2098 at 9/1/16 5:16 PM:
------------------------------------------------------------
In our implementation we created:
{{public class LESS_PING extends PING
{
@Override
public void findMembers( List<Address> members, boolean initial_discovery, Responses responses )
{
// no discovery except for the initial request
if ( initial_discovery )
{
try
{
sendDiscoveryRequest( cluster_name, members );
}
catch ( InterruptedIOException ie )
{
;
}
catch ( InterruptedException ex )
{
;
}
catch ( Throwable ex )
{
log.error( Util.getMessage( "FailedSendingDiscoveryRequest" ), ex );
}
}
}
protected void sendDiscoveryResponse(Address logical_addr, PhysicalAddress physical_addr,
String logical_name, final Address sender, boolean coord) {
final PingData data=new PingData(logical_addr, is_server, logical_name, physical_addr).coord(coord);
final Message rsp_msg=new Message(sender).setFlag(Message.Flag.INTERNAL, Message.Flag.OOB,
Message.Flag.DONT_BUNDLE)
.putHeader(this.id, new PingHeader(PingHeader.GET_MBRS_RSP)).setBuffer(marshal(data));
if(stagger_timeout > 0) {
int view_size=view != null? view.size() : 10;
int rank=Util.getRank(view, local_addr); // returns 0 if view or local_addr are null
if( rank > 5 )
{
// there will be plenty of responders
log.trace("%s: received GET_MBRS_REQ from %s, skipping response due to rank %d > max allowed of %d",
local_addr, sender, rank, max_rank_for_response );
}
else
{
long sleep_time = rank == 0 ? Util.random( stagger_timeout )
: stagger_timeout * rank / view_size - (stagger_timeout / view_size);
timer.schedule( new Runnable()
{
@Override
public void run()
{
log.trace( "%s: received GET_MBRS_REQ from %s, sending staggered response %s", local_addr, sender,
data );
down_prot.down( new Event( Event.MSG, rsp_msg ) );
}
}, sleep_time, TimeUnit.MILLISECONDS );
}
}
else
{
log.trace( "%s: received GET_MBRS_REQ from %s, sending response %s", local_addr, sender, data );
down_prot.down( new Event( Event.MSG, rsp_msg ) );
}
}
}
}}
This in effect limits discovery to initial discovery. In addition, discovery only responds if the rank < 5. Both of these changes work together to most eliminate discovery from the channel (because we have the IP address in the UUID)
was (Author: ndillman):
In our implementation we created:
public class LESS_PING extends PING
{
@Override
public void findMembers( List<Address> members, boolean initial_discovery, Responses responses )
{
// no discovery except for the initial request
if ( initial_discovery )
{
try
{
sendDiscoveryRequest( cluster_name, members );
}
catch ( InterruptedIOException ie )
{
;
}
catch ( InterruptedException ex )
{
;
}
catch ( Throwable ex )
{
log.error( Util.getMessage( "FailedSendingDiscoveryRequest" ), ex );
}
}
}
protected void sendDiscoveryResponse(Address logical_addr, PhysicalAddress physical_addr,
String logical_name, final Address sender, boolean coord) {
final PingData data=new PingData(logical_addr, is_server, logical_name, physical_addr).coord(coord);
final Message rsp_msg=new Message(sender).setFlag(Message.Flag.INTERNAL, Message.Flag.OOB,
Message.Flag.DONT_BUNDLE)
.putHeader(this.id, new PingHeader(PingHeader.GET_MBRS_RSP)).setBuffer(marshal(data));
if(stagger_timeout > 0) {
int view_size=view != null? view.size() : 10;
int rank=Util.getRank(view, local_addr); // returns 0 if view or local_addr are null
if( rank > 5 )
{
// there will be plenty of responders
log.trace("%s: received GET_MBRS_REQ from %s, skipping response due to rank %d > max allowed of %d",
local_addr, sender, rank, max_rank_for_response );
}
else
{
long sleep_time = rank == 0 ? Util.random( stagger_timeout )
: stagger_timeout * rank / view_size - (stagger_timeout / view_size);
timer.schedule( new Runnable()
{
@Override
public void run()
{
log.trace( "%s: received GET_MBRS_REQ from %s, sending staggered response %s", local_addr, sender,
data );
down_prot.down( new Event( Event.MSG, rsp_msg ) );
}
}, sleep_time, TimeUnit.MILLISECONDS );
}
}
else
{
log.trace( "%s: received GET_MBRS_REQ from %s, sending response %s", local_addr, sender, data );
down_prot.down( new Event( Event.MSG, rsp_msg ) );
}
}
}
This in effect limits discovery to initial discovery. In addition, discovery only responds if the rank < 5. Both of these changes work together to most eliminate discovery from the channel (because we have the IP address in the UUID)
> Discovery: reduce messages when IpAddressUUID is used
> -----------------------------------------------------
>
> Key: JGRP-2098
> URL: https://issues.jboss.org/browse/JGRP-2098
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0
>
>
> Since IpAddressUUID already contains the physical address, we don't need to exchange physical addresses in the discovery phase.
> Investigate whether this leads to reduced messaging in discovery, ie. only the coords might send a response. Once the new member has the view, it automatically knows the IP addresses and ports of all members, as the addresses in the view are of type IpAddressUUID.
> Also investigate whether address:logical_name associations should be done in a separate protocol, e.g. NAMING.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (JGRP-2098) Discovery: reduce messages when IpAddressUUID is used
by Neal Dillman (JIRA)
[ https://issues.jboss.org/browse/JGRP-2098?page=com.atlassian.jira.plugin.... ]
Neal Dillman commented on JGRP-2098:
------------------------------------
In our implementation we created:
public class LESS_PING extends PING
{
@Override
public void findMembers( List<Address> members, boolean initial_discovery, Responses responses )
{
// no discovery except for the initial request
if ( initial_discovery )
{
try
{
sendDiscoveryRequest( cluster_name, members );
}
catch ( InterruptedIOException ie )
{
;
}
catch ( InterruptedException ex )
{
;
}
catch ( Throwable ex )
{
log.error( Util.getMessage( "FailedSendingDiscoveryRequest" ), ex );
}
}
}
protected void sendDiscoveryResponse(Address logical_addr, PhysicalAddress physical_addr,
String logical_name, final Address sender, boolean coord) {
final PingData data=new PingData(logical_addr, is_server, logical_name, physical_addr).coord(coord);
final Message rsp_msg=new Message(sender).setFlag(Message.Flag.INTERNAL, Message.Flag.OOB,
Message.Flag.DONT_BUNDLE)
.putHeader(this.id, new PingHeader(PingHeader.GET_MBRS_RSP)).setBuffer(marshal(data));
if(stagger_timeout > 0) {
int view_size=view != null? view.size() : 10;
int rank=Util.getRank(view, local_addr); // returns 0 if view or local_addr are null
if( rank > 5 )
{
// there will be plenty of responders
log.trace("%s: received GET_MBRS_REQ from %s, skipping response due to rank %d > max allowed of %d",
local_addr, sender, rank, max_rank_for_response );
}
else
{
long sleep_time = rank == 0 ? Util.random( stagger_timeout )
: stagger_timeout * rank / view_size - (stagger_timeout / view_size);
timer.schedule( new Runnable()
{
@Override
public void run()
{
log.trace( "%s: received GET_MBRS_REQ from %s, sending staggered response %s", local_addr, sender,
data );
down_prot.down( new Event( Event.MSG, rsp_msg ) );
}
}, sleep_time, TimeUnit.MILLISECONDS );
}
}
else
{
log.trace( "%s: received GET_MBRS_REQ from %s, sending response %s", local_addr, sender, data );
down_prot.down( new Event( Event.MSG, rsp_msg ) );
}
}
}
This in effect limits discovery to initial discovery. In addition, discovery only responds if the rank < 5. Both of these changes work together to most eliminate discovery from the channel (because we have the IP address in the UUID)
> Discovery: reduce messages when IpAddressUUID is used
> -----------------------------------------------------
>
> Key: JGRP-2098
> URL: https://issues.jboss.org/browse/JGRP-2098
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0
>
>
> Since IpAddressUUID already contains the physical address, we don't need to exchange physical addresses in the discovery phase.
> Investigate whether this leads to reduced messaging in discovery, ie. only the coords might send a response. Once the new member has the view, it automatically knows the IP addresses and ports of all members, as the addresses in the view are of type IpAddressUUID.
> Also investigate whether address:logical_name associations should be done in a separate protocol, e.g. NAMING.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (WFCORE-952) Use WildFly Common for null param checks
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-952?page=com.atlassian.jira.plugin... ]
Brian Stansberry commented on WFCORE-952:
-----------------------------------------
Sure, or just feel free to schedule it again for whenever. As part of releasing Alpha7 I went through all the JIRAs scheduled for it that just seemed to be rolling from release to release and tried to clean some of those out. My previous comment is generic one that I pasted into all of them.
> Use WildFly Common for null param checks
> ----------------------------------------
>
> Key: WFCORE-952
> URL: https://issues.jboss.org/browse/WFCORE-952
> Project: WildFly Core
> Issue Type: Task
> Reporter: David Lloyd
> Priority: Minor
>
> For each module, do the following:
> * Locate any/all null param check methods in the log msg
> * Replace them with calls to org.wildfly.common.Assert#checkNotNullParam or related method as needed
> * Replace the old null param check method with a comment that reserves the ID and shows that it was previously used for that purpose
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:28 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- we get the session corresponding to the session id from the ISPN cache (StatefulComponentInstanceInterceptor)
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:25 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:24 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code:xml}
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:23 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
Here is the cache configuration from the server configuration:
{code:xml}
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
{code:xml}
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:19 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be? Unless the numOwners=1, should it not look elsewhere for a copy of the cache entry and return that as a result of the invocation, rather than throw an exception if the owner node is not available?
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be?
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0
[JBoss JIRA] (WFLY-7029) RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-7029?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-7029 at 9/1/16 4:15 PM:
-------------------------------------------------------------------
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Note that despite the fact that clean shutdown was used on node-0 with a timeout of 15 seconds, it still allows node-0 to shutdown fairly quickly, as there are probably few pending invocations at the time of shutdown which the suspend mechanism needs to wait for to complete.
Need to check:
a) when is the topology change received by node-1?
-- hard to tell, I don't see any topology change messages after node-0 stops; only after it has restarted
-- however, the JGroups view change is is visible and it is received after the exception arises
4:31:43:502 -> exception is raised
4:31:43:732 -> JGroups view change arrives
-- I assume that the reason why node-1 sends the RPC to node-0 is because it is using the latest (but out of date) topoology that it has on hand
b) Is this simply a case that it is impossible for ISPN to react to topology changes as quickly as invocations may get redirected by the EJB client (i.e. in a few ms)?
If that is the case, i'm wondering what to do in general in the case where an owner node is no longer available in the current topology (as is the case here with node-0 and the session instance it is holding) despite the fact that the current topology says it should be?
was (Author: rachmato):
Just a summary of the circumstances:
- cluster of node-0 and node-1, with SFSB SlowToDestroyStatefulIncrementorBean deployed on both nodes
- a session is created on node-0, which is the owner node
- invocations progress normally against node-0
- on node-0, the process of undeploying the deployment and shutting down the node is started
- an invocation arrives at node-0 as the deployment is undeploying
-- the EJB interceptor returns a EJBComponentUnavailable exception to the EJB client for the invocation
-- the invocation is retried on node-1
- the invocation proceeds on node-1, within a container managed transaction (CMTTxInterceptor)
- in one EJB interceptor (???? - forgot), we get the session corresponding to the session id from the ISPN cache
- a GetKeyValueCommand is invoked on node-1 and this causes an RPC to get the session from its owner on node-0
- the RPC arrives at node-0, but as the cache is in stopping mode, the exception of this issue arises
- the ISPN caller of get() receives a RemoteException
- the CMTTxInterceptor fails the invocation by rolling back and returns an exception to the client
Need to check:
- when is the topology change received by node-1?
- what to do in case an owner node is no longer available?
> RemoteFailoverTestCast#testGracefulShutdownConcurrentFailover fails with "ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations."
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-7029
> URL: https://issues.jboss.org/browse/WFLY-7029
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Richard Achmatowicz
>
> https://ci.wildfly.org/viewLog.html?buildId=24851&tab=buildResultsDiv&bui...
> {noformat}
> 04:31:43,323 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopping container=container-0, timeout=15
> 04:31:43,324 INFO [org.jboss.as.arquillian.container.controller.ClientWildFlyContainerController] (main) Manual stopping of a server instance with timeout=15
> &#27;[0m04:31:43,338 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 15000 ms timeout.
> &#27;[0m04:31:43,343 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-10) Retrying invocation which failed on node node-0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=remote-failover-test,distinctname=,beanname=SlowToDestroyStatefulIncrementorBean,viewclassname=org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor]
> at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
> at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386)
> at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m04:31:43,345 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> &#27;[0m&#27;[0m04:31:43,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
> &#27;[0m&#27;[0m04:31:43,436 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 29) MODCLUSTER000002: Initiating mod_cluster shutdown
> &#27;[0m&#27;[0m04:31:43,445 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel hibernate
> &#27;[0m&#27;[0m04:31:43,459 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel hibernate
> &#27;[0m&#27;[0m04:31:43,463 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel server
> &#27;[0m&#27;[0m04:31:43,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel server
> &#27;[0m&#27;[0m04:31:43,471 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk
> &#27;[0m&#27;[0m04:31:43,472 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 0 milliseconds
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
> &#27;[0m&#27;[0m04:31:43,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
> &#27;[0m&#27;[0m04:31:43,502 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel testDbPersistence
> &#27;[0m&#27;[0m04:31:43,505 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel testDbPersistence
> &#27;[0m&#27;[33m04:31:43,502 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (thread-11,ee,node-0) ISPN000071: Caught exception when handling command LockControlCommand{cache=remote-failover-test.jar, keys=[UnknownSessionID [6951504948515351686656515249505365655750686851525048706670486766]], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction:<node-1>:48:remote}: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
> &#27;[0m&#27;[0m04:31:43,517 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> &#27;[0m&#27;[0m04:31:43,518 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> &#27;[0m&#27;[0m04:31:43,535 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> &#27;[0m&#27;[0m04:31:43,547 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 2) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> &#27;[0m04:31:43,565 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-14) EJBCLIENT000016: Channel Channel ID b007a55b (outbound) of Remoting connection 017c35ab to /0:0:0:0:0:0:0:1:8080 of endpoint "config-based-ejb-client-endpoint" <1c33eee> can no longer process messages
> &#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
> &#27;[0m&#27;[0m04:31:43,576 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel ejb
> &#27;[0m&#27;[0m04:31:43,575 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow AJP listener ajp suspending
> &#27;[0m&#27;[0m04:31:43,591 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to [::1]:8009
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTPS listener https suspending
> &#27;[0m&#27;[0m04:31:43,592 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to [::1]:8443
> &#27;[0m&#27;[31m04:31:43,585 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-11) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> &#27;[0m&#27;[31m04:31:43,588 ERROR [org.jboss.as.ejb3.invocation] (default task-11) WFLYEJB0034: EJB Invocation failed on component SlowToDestroyStatefulIncrementorBean for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment(): javax.ejb.EJBException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:328)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:263)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
> at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-0, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:380)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:260)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.MPING.up(MPING.java:178)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> ... 1 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:106)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:205)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:175)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
> at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
> ... 27 more
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: remote-failover-test.jar is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:364)
> at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:354)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:138)
> at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:38)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.createContext(PendingTxAction.java:107)
> at org.infinispan.remoting.inboundhandler.action.PendingTxAction.init(PendingTxAction.java:57)
> at org.infinispan.remoting.inboundhandler.action.BaseLockingAction.check(BaseLockingAction.java:38)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:42)
> at org.infinispan.remoting.inboundhandler.action.DefaultReadyAction.isReady(DefaultReadyAction.java:45)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler$1.isReady(NonTotalOrderTxPerCacheInboundInvocationHandler.java:124)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:51)
> at org.infinispan.executors.LazyInitializingBlockingTaskAwareExecutorService.execute(LazyInitializingBlockingTaskAwareExecutorService.java:46)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:130)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
> ... 33 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
- 1
- 0