[jboss-jira] [JBoss JIRA] (WFLY-5178) Intermittent failure: NullPointerException during undeployment
Richard Achmatowicz (JIRA)
issues at jboss.org
Wed Nov 11 12:49:00 EST 2015
[ https://issues.jboss.org/browse/WFLY-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127917#comment-13127917 ]
Richard Achmatowicz commented on WFLY-5178:
-------------------------------------------
FYI: This is a repeat of the issue with RemoteFailoverTestCase in the Wildfly testsuite, where we have problems with EJBClient exceptions during undeployment of an application. So this is in some sense a duplicate.
When we deploy an application which makes use of EJBs, we create an entry in the deployment repository to represent the deployment and its EJB components. Also, for each application which contains a SFSB, we create a cache which backs sessions for that SFSB component. When an EJBClient message arrives at a server, the deployment repository is used to lookup the component mentioned in the EJBClient message and get access to the required components.
In the case of an EJBClient invocation request, after the message is unpacked, we call findEJB(...) to locate the EJB component to which the invocation applies. If that component is found in the deployment repository(*), an instance of the component's interface is obtained, and the invocation starts to proceed through the interface's interceptors. One of these is the EJBSuspendInterceptor, which will check to see if we the server is in the process of being suspended (part of clean shutdown); if we are, then the invocation will return an exception indicating this. This interceptor only has effect during shutown, and not during undeploy. After the EJBSuspendInterceptor comes the ShutdownInterceptor. This interceptor is controlled by EJBComponent.start() and EJBComponent.stop(). When EJBComponent.stop() is called, this causes the interceptor to return an exception indicating that the component is shutting down. This interceptor is triggered during undeployment of a component, as well as clean shutdown (because the component will be undeployed as part of shutdown).
In the logs, as the undeploy takes place, you can see the effects of the ShutdownInterceptor on EJBClient method invocations:
{noformat}
[JBossINF] [31m23:52:48,178 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 5) WFLYEJB0034: EJB Invocation failed on component RemoteStatefulSBImpl for method public abstract int org.jboss.test.clusterbench.common.ejb.CommonStatefulSB.getSerialAndIncrement(): org.jboss.as.ejb3.component.EJBComponentUnavailableException: WFLYEJB0421: Invocation cannot proceed as component is shutting down
[JBossINF] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:59)
[JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
[JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
[JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
[JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
[JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
[JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
[JBossINF] at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
[JBossINF] at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
[JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
[JBossINF] at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
[JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
[JBossINF] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
[JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:331)
[JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69)
[JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202)
[JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{noformat}
In the case of EJB message requests to create a new session, the situation is different. The message request comes in, is unpacked, and then the SessionOpenRequestHandler is called to create a new session and return the session id. This method looks up the EJB component in the deploymentRepository using findEJB(...) as before, but this time, instead of looking up the component's interface and running through the interceptors, it calls a method on the SFSB to create a new cache entry for the session. The method is StatefulSessionComponent.createSessionRemote(). This method also includes logic to check to see of the server is being suspended, and will generate an exception if it is, just as in the case of EjbSuspendInterceptor. However, in the case of undeployment, this method does nothing in particular. It doesn't receive the protection that ShutdownInterceptor provides. This means that even if the application is being undeployed, the request processing will still get to call cache.create().getSessionId() on the cache instance backing the session bean's session instances, whether that cache has been stopped by the time the invocation arrives or not. This is the problem. The exceptions seen in the log reflect this fact:
{noformat}
[JBossINF] [31m23:52:48,224 ERROR [org.jboss.as.ejb3.remote] (EJB default - 6) WFLYEJB0151: Exception while generating session id for component RemoteStatefulSBImpl with invocation id: 5942 on channel Channel ID 6309fce9 (inbound) of Remoting connection 6dfa1b19 to /10.16.90.52:41631: javax.ejb.EJBException: java.lang.NullPointerException
[JBossINF] at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSessionRemote(StatefulSessionComponent.java:263)
[JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.SessionOpenRequestHandler$SessionIDGeneratorTask.run(SessionOpenRequestHandler.java:145)
[JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
[JBossINF] Caused by: java.lang.NullPointerException
[JBossINF] at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSession(StatefulSessionComponent.java:238)
[JBossINF] at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSessionRemote(StatefulSessionComponent.java:258)
[JBossINF] ... 7 more
{noformat}
In fact, these two exceptions above are right next to each other in the logs.
(*) it can and does happen that if an EJBClient request arrives very late, after shutdown has progressed considerably, we can get NPEs on the lookups in the deployment repository.
> Intermittent failure: NullPointerException during undeployment
> --------------------------------------------------------------
>
> Key: WFLY-5178
> URL: https://issues.jboss.org/browse/WFLY-5178
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 10.0.0.Beta1
> Reporter: Michal Vinkler
> Assignee: Richard Achmatowicz
> Labels: clean_undeploy
>
> Intermittent failure, so far seen in only one failover test for remote stateful EJBs:
> scenario ejb-ejbremote-undeploy-dist-async
> Setup: 4 node cluster, one node at time undeploys application (clusterbench-ee7.ear), while standalone clients keep calling the application.
> Just before server perf19 finished undeployment of the application, it logged this error 50 times:
> {code}
> [JBossINF] [0m[31m14:50:41,136 ERROR [org.jboss.as.ejb3.remote] (EJB default - 9) WFLYEJB0151: Exception while generating session id for component RemoteStatefulSBImpl with invocation id: -28551 on channel Channel ID 1ca843e6 (inbound) of Remoting connection 4b2907f9 to /10.16.90.52:58920: javax.ejb.EJBException: java.lang.NullPointerException
> [JBossINF] at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSessionRemote(StatefulSessionComponent.java:263)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.SessionOpenRequestHandler$SessionIDGeneratorTask.run(SessionOpenRequestHandler.java:145)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> [JBossINF] Caused by: java.lang.NullPointerException
> [JBossINF] at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSession(StatefulSessionComponent.java:238)
> [JBossINF] at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSessionRemote(StatefulSessionComponent.java:258)
> [JBossINF] ... 7 more
> {code}
> The other nodes did not log this error.
> Server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbremote-undeploy-dist-async/4/console-perf19/
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list