2:10 p.m.
We have an MDB deployed in jboss 4.0.5 container. This code has been in production for
3+years and has never had an issue, until today.
For some reason the MDB failed. I don't think it was actually the MDB as the stack
trace indicates it is an error in the JMSContainerInvoker.getMetaData class/method.
The symptoms were that the MDB just stopped processing messages from it's associated
queue. The queue was still accepting messages (the load on this queue processes 6-8K
messages per day) and we could see the count on the queue increase through the jmx-console
while we were trying to figure out what was wrong.
When we went to inspect the MDB it's state was failure. Upon attempting to restart
delivery we got this error
| 2008-03-21 10:59:47,405 WARN [org.jboss.system.ServiceController] Problem starting
service
jboss.j2ee:service=EJB,plugin=invoker,binding=message-driven-bean,jndiName=local/XMLMerchantProcess@711460
| java.lang.NullPointerException
| at
org.jboss.ejb.plugins.jms.JMSContainerInvoker.getMetaData(JMSContainerInvoker.java:264)
| at
org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl.handleFailure(JMSContainerInvoker.java:1337)
| at
org.jboss.ejb.plugins.jms.JMSContainerInvoker.startService(JMSContainerInvoker.java:845)
| at
org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
| at
org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at
org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
| at $Proxy0.start(Unknown Source)
| at org.jboss.system.ServiceController.start(ServiceController.java:417)
| at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:194)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258)
| at org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)
| at
org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:262)
| at
org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)
| at
org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
| at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
| at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
| at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
| at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
| at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
| at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
| at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
| at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
| at
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
| at
org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:419)
| at
org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495)
| at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
| at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
| at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
| at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
| at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
| at
org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
| at java.lang.Thread.run(Thread.java:595)
|
|
|
We then decided to do something more drastic and tried to destroy and recreate the bean.
After creation the bean's state was stopped and everything looked good in the
jmx-console. When we attempted to start delivery on the bean again it threw the exact same
exception and the state changed to fail once again.
We then tried a hot redeploy of the MDB in the container (moved the jar out of deploy and
back into deploy) This made no difference in the outcome and produced the exact same stack
trace.
I scoured this forum (and the internet in general) for any solution but the only one I
found was to take the exact same actions we had already taken. That was to restart the
bean with (startDelivery method on mbean) which kept throwing the above error.
Since the volume here is large enough that people were starting to complain that the
messages were not being processed we tried the only other solution we could think of,
which was to restart the entire app server, which really should not be the solution in my
opinion. Once restarted it processed the 2300+ messages backed up in the queue withing 30
or so minutes and is currently running Ok.
This code has been really stable historically as the app server almost never gets shut
down. The last restart of the server we had was on 3/12/08 when we deployed code for
another application (our admins have a bounce policy when deploying new code for some
reason). The timestamp on the jar for this MDB is 11/18/2006 so it has been in production
for almost 18 months without having any problems.
I guess my question would ultimately be:
Am I running into a 4.0.5 bug and need to upgrade my server?
The stack trace seems to indicate that the MDB's metadata has been dropped from the
container. Why undeploying and redeploying would not correct this problem is probably a
question for the developers.
Thaks for any info/insight anyone can provide.
Andre
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4138363#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...