[ https://issues.jboss.org/browse/WFLY-5304?page=com.atlassian.jira.plugin.... ] Richard Achmatowicz edited comment on WFLY-5304 at 3/3/16 3:08 PM: ------------------------------------------------------------------- After discussion with Paul, its should the the case that the three services involved: - EndpointService (which receives EJB-related messages on the Remoting channel jboss.ejb) - EJBRemoteConnectorService (which processes these as invocations) - DeploymentRepository (which allows locating deployments) should have dependencies in place such that the DeploymentRepository and EndpointService do not stop before the EJBRemoteConnectorService stops. This seems to be the case when I instrument the build with some trace logging and introduce delays in stopping: {noformat} 13:30:54,767 INFO [stdout] (MSC service thread 1-3) org.jboss.as.remoting.EndpointService(lenovo): stop() called for endpoint lenovo:MANAGEMENT - begin 13:30:54,768 INFO [stdout] (MSC service thread 1-1) org.jboss.as.ejb3.remote.EJBRemoteConnectorService(lenovo): stop() called - begin 13:30:54,768 INFO [stdout] (MSC service thread 1-1) org.jboss.as.ejb3.remote.EJBRemoteConnectorService(lenovo): stop() called - sleeping 13:30:54,769 INFO [stdout] (MSC service thread 1-3) org.jboss.as.remoting.EndpointService(lenovo): stop() called for endpoint lenovo:MANAGEMENT - end 13:30:57,768 INFO [stdout] (MSC service thread 1-1) org.jboss.as.ejb3.remote.EJBRemoteConnectorService(lenovo): stop() called - end 13:30:57,769 INFO [stdout] (MSC service thread 1-3) org.jboss.as.ejb3.deployment.DeploymentRepository{lenovo): stop() called - begin 13:30:57,769 INFO [stdout] (MSC service thread 1-6) org.jboss.as.remoting.EndpointService(lenovo): stop() called for endpoint lenovo - begin 13:30:57,770 INFO [stdout] (MSC service thread 1-3) org.jboss.as.ejb3.deployment.DeploymentRepository(lenovo): stop() called - end 13:30:57,770 INFO [stdout] (MSC service thread 1-6) org.jboss.as.remoting.EndpointService(lenovo): stop() called for endpoint lenovo - end {noformat} So, it seems that it should be the case that the DeploymentRepository should be available as long as The EJBRemoteConnectorService is available. However, there seems to be no way of preventing this scenario: - a message arrives at the Remoting endpont - the message M gets handled by the VersionOneProtocolChannelReceiver and processing in that handler starts in a separate thread - shutdown is called - EJBRemoteConnectorService calls stop, which unregisters the Endpoint jboss.ejb and the message handler - DeploymentRepository calls stop - the processing of the message M continues - the DeploymentRepository is no longer available and NPE results This scenario is not unlikely - in our shutdown tests, shutdown starts when the server is under load. In other words, for invocations which have arrived and have started processing but have not yet reached the interceptor stage, in the presence of a concurrent shutdown request, there is no mechanism to prevent the three components above from shutting down before the initial stages of message processing have completed. ControlPoints are used at the EJB interceptor level as they can (i) block access to specific EJB interfaces and (ii) count outstanding requests (those which have passed through the interceptor but have not yet returned). So using a ControlPoint for this issue here might not be appropriate. Also, during suspend, the DeploymentRepository will always be available any way. Here, the problem is that once shutdown has started, the EJB service is no longer guaranteed to be available. What we need is a way to check when the server is shutting down, so that when doing the initial EJB request processing, we can just return null in the lookup of finfdEJB(). was (Author: rachmato): After discussion with Paul, its should the the case that the three services involved: - EndpointService (which receives EJB-related messages on the Remoting channel jboss.ejb) - EJBRemoteConnectorService (which processes these as invocations) - DeploymentRepository (which allows locating deployments) should have dependencies in place such that the DeploymentRepository and EndpointService do not stop before the EJBRemoteConnectorService stops. This seems to be the case when I instrument the build with some trace logging and introduce delays in stopping: {noformat} 13:30:54,767 INFO [stdout] (MSC service thread 1-3) org.jboss.as.remoting.EndpointService(lenovo): stop() called for endpoint lenovo:MANAGEMENT - begin 13:30:54,768 INFO [stdout] (MSC service thread 1-1) org.jboss.as.ejb3.remote.EJBRemoteConnectorService(lenovo): stop() called - begin 13:30:54,768 INFO [stdout] (MSC service thread 1-1) org.jboss.as.ejb3.remote.EJBRemoteConnectorService(lenovo): stop() called - sleeping 13:30:54,769 INFO [stdout] (MSC service thread 1-3) org.jboss.as.remoting.EndpointService(lenovo): stop() called for endpoint lenovo:MANAGEMENT - end 13:30:57,768 INFO [stdout] (MSC service thread 1-1) org.jboss.as.ejb3.remote.EJBRemoteConnectorService(lenovo): stop() called - end 13:30:57,769 INFO [stdout] (MSC service thread 1-3) org.jboss.as.ejb3.deployment.DeploymentRepository{lenovo): stop() called - begin 13:30:57,769 INFO [stdout] (MSC service thread 1-6) org.jboss.as.remoting.EndpointService(lenovo): stop() called for endpoint lenovo - begin 13:30:57,770 INFO [stdout] (MSC service thread 1-3) org.jboss.as.ejb3.deployment.DeploymentRepository(lenovo): stop() called - end 13:30:57,770 INFO [stdout] (MSC service thread 1-6) org.jboss.as.remoting.EndpointService(lenovo): stop() called for endpoint lenovo - end {noformat} So, it seems that it should be the case that the DeploymentRepository should be available as long as The EJBRemoteConnectorService is available. However, there seems to be no way of preventing this scenario: - a message arrives at the Remoting endpont - the message M gets handled by the VersionOneProtocolChannelReceiver and processing in that handler starts in a separate thread - shutdown is called - EJBRemoteConnectorService calls stop, which unregisters the Endpoint jboss.ejb and the message handler - DeploymentRepository calls stop - the processing of the message M continues - the DeploymentRepository is no longer available and NPE results This scenario is not unlikely - in our shutdown tests, shutdown starts when the server is under load. In other words, for invocations which have arrived and have started processing but have not yet reached the interceptor stage, in the presence of a concurrent shutdown request, there is no mechanism to prevent the three components above from shutting down before the initial stages of message processing have completed. In other words, we could probably use a ControlPoint at this point in EJB request processing rather than later on during interceptor processing. A good spot might be in the findEJB() method which is the first place the DeploymentRepository is accessed and which could easily return a null indicating that the EJB service as a whole (and not just one specific EJB application, as addressed by the suspend interceptor) is not available. -- This message was sent by Atlassian JIRA (v6.4.11#64026)