[ https://issues.jboss.org/browse/WFLY-4234?page=com.atlassian.jira.plugin.... ] Richard Achmatowicz updated WFLY-4234: -------------------------------------- Description: The scenario is performing undeployment of an EJB application in the presence of incoming requests. If my understanding of the suspend/resume mechanism is correct, in the ideal case, performing an undeployment should disable invocations arriving after undeployment has been initiated, and also allow requests active at undeployment initiation some time to complete. In the clustering integration test RemoteFailoverTestCase, I am seeing failures in which the foloowing sequence of events occurs: - an EJB invocation on EJB SFSB X gets past the EJBSuspendInterceptor - undeployment is initiated and suspend is called on the ControlPoint for X - undeployment proceeds, removing certain resources required for the invocation - the active invocation fails due to resources not being available Here is one such example (I have added some of my own tracing to identify ControlPoint activities.). This is a log from the test in which invocations are arriving from an EJB client during undeployment. The request with input value 2 starts and at about the same time, undeployment is called. The invocation gets past the interceptor but never gets a chance to finish. {code} 14:22:04,459 INFO [org.jboss.ejb.client] (main) JBoss EJB Client version 2.0.1.Final 14:22:04,485 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-5) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] 14:22:04,492 INFO [org.jboss.ejb.client.remoting] (main) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@2ba6ac4c, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)4b79c7f7,channel=jboss.ejb,nodename=node-0]} on channel Channel ID ad5ee733 (outbound) of Remoting connection 29cd434b to /127.0.0.1:8080 14:22:04,500 INFO [stdout] (EJB default - 1) ControlPoint: beginRequest called 14:22:04,529 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-15) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] 14:22:04,530 INFO [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-3-thread-2) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@2ba6ac4c, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)74f6ab8a,channel=jboss.ejb,nodename=node-1]} on channel Channel ID d24c9830 (outbound) of Remoting connection 759d3b52 to /127.0.0.1:8180 14:22:04,542 INFO [org.jboss.ejb.client] (EJB default - 1) JBoss EJB Client version 2.0.1.Final 14:22:04,560 INFO [stdout] (EJB default - 1) ControlPoint: requestComplete called 14:22:09,682 INFO [stdout] (EJB default - 2) ControlPoint: beginRequest called 14:22:09,702 INFO [stdout] (EJB default - 2) ControlPoint: requestComplete called CLIENT: making invocation on bean: value = 1 14:22:09,710 INFO [stdout] (EJB default - 3) ControlPoint: beginRequest called 14:22:09,715 INFO [stdout] (EJB default - 3) ControlPoint: requestComplete called CLIENT: made invocation on bean [ node-0 ]: new value = 2 14:22:09,717 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Undeploying deployment=deployment-0 CLIENT: making invocation on bean: value = 2 14:22:09,728 INFO [stdout] (EJB default - 4) ControlPoint: beginRequest called 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 29) EJBComponent: stopping remote-failover-test:remote-failover-test:SlowToDestroyStatefulIncrementorBean 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 31) EJBComponent: stopping remote-failover-test:remote-failover-test:StatefulIncrementorBean 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 67) EJBComponent: stopping remote-failover-test:remote-failover-test:SecureStatelessIncrementorBean 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 31) ControlPoint: pause called 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 4) EJBComponent: stopping remote-failover-test:remote-failover-test:StatelessIncrementorBean 14:22:09,731 INFO [stdout] (ServerService Thread Pool -- 29) ControlPoint: pause called 14:22:09,731 INFO [stdout] (ServerService Thread Pool -- 11) EJBComponent: stopping remote-failover-test:remote-failover-test:ClientIncrementorBean 14:22:09,731 INFO [stdout] (ServerService Thread Pool -- 4) ControlPoint: pause called 14:22:09,732 INFO [stdout] (ServerService Thread Pool -- 67) ControlPoint: pause called 14:22:09,732 INFO [stdout] (ServerService Thread Pool -- 11) ControlPoint: pause called 14:22:09,733 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk 14:22:09,735 INFO [stdout] (EJB default - 4) ControlPoint: requestComplete called 14:22:09,735 ERROR [org.jboss.as.ejb3] (EJB default - 4) WFLYEJB0150: Could not write method invocation failure for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment() on bean named SlowToDestroyStatefulIncrementorBean for appname modulename remote-failover-test distinctname due to: java.lang.NullPointerException at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.getWeakAffinity(MethodInvocationMessageHandler.java:269) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$500(MethodInvocationMessageHandler.java:69) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:237) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) at org.jboss.threads.JBossThread.run(JBossThread.java:320) 14:22:09,736 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 2 milliseconds 14:22:09,736 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-5) EJBCLIENT000016: Channel Channel ID ad5ee733 (outbound) of Remoting connection 29cd434b to /127.0.0.1:8080 can no longer process messages 14:22:09,738 INFO [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0010: Stopping weld service for deployment remote-failover-test.jar 14:22:09,740 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container 14:22:09,760 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk 14:22:09,761 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 1 entries in 0 milliseconds 14:22:09,762 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment remote-failover-test.jar (runtime-name: remote-failover-test.jar) in 36ms 14:22:09,764 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped dist cache from ejb container 14:22:09,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000080: Disconnecting JGroups channel ejb 14:22:09,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000082: Stopping the RpcDispatcher for channel ejb 14:22:09,772 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-4,ee,node-1) ISPN000094: Received new cluster view for channel ejb: [node-1|2] (1) [node-1] 14:22:09,807 INFO [org.jboss.as.repository] (management-handler-thread - 1) WFLYDR0002: Content removed from location /home/nrla/projects/wildfly-git-repo/testsuite/integration/clustering/target/wildfly-SYNC-tcp-0/standalone/data/content/66/642dea2bc32c3ba888dd173db2b91d18b848b0/content 14:22:09,807 INFO [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0009: Undeployed "remote-failover-test.jar" (runtime-name: "remote-failover-test.jar") {code} Looking at the use of ControlPoints within EJBs, it seems that on undeployment, the EJBComponent.stop() method calls ControlPoint.pause() only which will cause the interceptor to block any future invocations but also not prevent the undeployment from continuing, despite the fact that a request is active. In this case, it seems that once the EJB components have been stopped, the cache entries for their state have been passivated before the invocation has completed. Do we need more mechanism for such cases? was: The scenario is performing undeployment of an EJB application in the presence of incoming requests. If my understanding of the suspend/resume mechanism is correct, in the ideal case, performing an undeployment should disable invocations arriving after undeployment has been initiated, and also allow requests active at undeployment initiation some time to complete. In the clustering integration test RemoteFailoverTestCase, I am seeing failures in which the foloowing sequence of events occurs: - an EJB invocation on EJB SFSB X gets past the EJBSuspendInterceptor - undeployment is initiated and suspend is called on the ControlPoint for X - undeployment proceeds, removing certain resources required for the invocation - the active invocation fails due to resources not being available Here is one such example (I have added some of my own tracing to identify ControlPoint activities.). This is a log from the test in which invocations are arriving from an EJB client during undeployment. The request with input value 2 starts and at about the same time, undeployment is called. The invocation gets past the interceptor but never gets a chance to finish. {code} 14:22:04,459 INFO [org.jboss.ejb.client] (main) JBoss EJB Client version 2.0.1.Final 14:22:04,485 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-5) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] 14:22:04,492 INFO [org.jboss.ejb.client.remoting] (main) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@2ba6ac4c, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)4b79c7f7,channel=jboss.ejb,nodename=node-0]} on channel Channel ID ad5ee733 (outbound) of Remoting connection 29cd434b to /127.0.0.1:8080 14:22:04,500 INFO [stdout] (EJB default - 1) ControlPoint: beginRequest called 14:22:04,529 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-15) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] 14:22:04,530 INFO [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-3-thread-2) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@2ba6ac4c, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)74f6ab8a,channel=jboss.ejb,nodename=node-1]} on channel Channel ID d24c9830 (outbound) of Remoting connection 759d3b52 to /127.0.0.1:8180 14:22:04,542 INFO [org.jboss.ejb.client] (EJB default - 1) JBoss EJB Client version 2.0.1.Final 14:22:04,560 INFO [stdout] (EJB default - 1) ControlPoint: requestComplete called 14:22:09,682 INFO [stdout] (EJB default - 2) ControlPoint: beginRequest called 14:22:09,702 INFO [stdout] (EJB default - 2) ControlPoint: requestComplete called CLIENT: making invocation on bean: value = 1 14:22:09,710 INFO [stdout] (EJB default - 3) ControlPoint: beginRequest called 14:22:09,715 INFO [stdout] (EJB default - 3) ControlPoint: requestComplete called CLIENT: made invocation on bean [ node-0 ]: new value = 2 14:22:09,717 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Undeploying deployment=deployment-0 CLIENT: making invocation on bean: value = 2 14:22:09,728 INFO [stdout] (EJB default - 4) ControlPoint: beginRequest called 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 29) EJBComponent: stopping remote-failover-test:remote-failover-test:SlowToDestroyStatefulIncrementorBean 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 31) EJBComponent: stopping remote-failover-test:remote-failover-test:StatefulIncrementorBean 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 67) EJBComponent: stopping remote-failover-test:remote-failover-test:SecureStatelessIncrementorBean 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 31) ControlPoint: pause called 14:22:09,730 INFO [stdout] (ServerService Thread Pool -- 4) EJBComponent: stopping remote-failover-test:remote-failover-test:StatelessIncrementorBean 14:22:09,731 INFO [stdout] (ServerService Thread Pool -- 29) ControlPoint: pause called 14:22:09,731 INFO [stdout] (ServerService Thread Pool -- 11) EJBComponent: stopping remote-failover-test:remote-failover-test:ClientIncrementorBean 14:22:09,731 INFO [stdout] (ServerService Thread Pool -- 4) ControlPoint: pause called 14:22:09,732 INFO [stdout] (ServerService Thread Pool -- 67) ControlPoint: pause called 14:22:09,732 INFO [stdout] (ServerService Thread Pool -- 11) ControlPoint: pause called 14:22:09,733 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk 14:22:09,735 INFO [stdout] (EJB default - 4) ControlPoint: requestComplete called 14:22:09,735 ERROR [org.jboss.as.ejb3] (EJB default - 4) WFLYEJB0150: Could not write method invocation failure for method public abstract org.jboss.as.test.clustering.cluster.ejb.remote.bean.Result org.jboss.as.test.clustering.cluster.ejb.remote.bean.Incrementor.increment() on bean named SlowToDestroyStatefulIncrementorBean for appname modulename remote-failover-test distinctname due to: java.lang.NullPointerException at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.getWeakAffinity(MethodInvocationMessageHandler.java:269) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$500(MethodInvocationMessageHandler.java:69) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:237) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) at org.jboss.threads.JBossThread.run(JBossThread.java:320) 14:22:09,736 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 2 entries in 2 milliseconds 14:22:09,736 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-5) EJBCLIENT000016: Channel Channel ID ad5ee733 (outbound) of Remoting connection 29cd434b to /127.0.0.1:8080 can no longer process messages 14:22:09,738 INFO [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0010: Stopping weld service for deployment remote-failover-test.jar 14:22:09,740 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped remote-failover-test.jar cache from ejb container 14:22:09,760 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000029: Passivating all entries to disk 14:22:09,761 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 29) ISPN000030: Passivated 1 entries in 0 milliseconds 14:22:09,762 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment remote-failover-test.jar (runtime-name: remote-failover-test.jar) in 36ms 14:22:09,764 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 29) WFLYCLINF0003: Stopped dist cache from ejb container 14:22:09,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000080: Disconnecting JGroups channel ejb 14:22:09,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000082: Stopping the RpcDispatcher for channel ejb 14:22:09,772 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-4,ee,node-1) ISPN000094: Received new cluster view for channel ejb: [node-1|2] (1) [node-1] 14:22:09,807 INFO [org.jboss.as.repository] (management-handler-thread - 1) WFLYDR0002: Content removed from location /home/nrla/projects/wildfly-git-repo/testsuite/integration/clustering/target/wildfly-SYNC-tcp-0/standalone/data/content/66/642dea2bc32c3ba888dd173db2b91d18b848b0/content 14:22:09,807 INFO [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0009: Undeployed "remote-failover-test.jar" (runtime-name: "remote-failover-test.jar") {code} Looking at the use of ControlPoints within EJBs, it seems that on undeployment, the EJBComponent.stop() method calls ControlPoint.pause() only which will cause the interceptor to block any future invocations but also not prevent the undeployment from continuing, despite the fact that a request is active. Do we need more mechanism for such cases? -- This message was sent by Atlassian JIRA (v6.3.11#6341)