[jboss-user] [JBoss Messaging] - Occasional NPE when use JGroups multiplexor
Jeremy Stone
do-not-reply at jboss.com
Fri Jan 18 09:39:17 EST 2008
When we start up our JBoss application we sometimes get the exception shown below. It occurs on roughly 1/4 of startups. We are running JBossAS 4.2.0.GA with JBoss Messaging 1.4.0.SP3 (also saw problem with SP2). We are running with clustered destinations (but with just one cluster node at present). We use JGroups 2.5.1 with the multiplexor using the udp-sync channel config (shown below) for the control channel.
Debugging, the following seems to happen at application server startup:
- Main thread calls GroupMember.start() which connects to the controlChannel
- Multiplexer thread calls back to the ControlMembershipListener viewAccepted() which creates the latch object
- Main thread continues to call latch.countDown(). There is clearly a race condition here which causes latch to be null sometimes. Bang!
Note:
- problem does not occur if dont use multiplexor in JBoss messaging config (i.e. use configuration bundled with JBM): i.e. callback to viewAccepted() is on the main thread.
- but it does occur if use multiplexor with the udp-sync channel config replaced by that in the configuration bundled with JBM. This differs from that in udp-sync (below) in that in the former use_flush is set for STATE_TRANSFER and GMS, and in udp-sync use_concurrent_stack=true with thread_pool and oob.thread_pool.enabled both =true. So these differences do not account for the problem.
Setting the system property jgroups.mux.enabled=false makes it work again (with the multiplexor-style configuration) and is our solution for the time being.
Any thoughts?
2008-01-17 19:02:33,118 37694 WARN [org.jboss.system.ServiceController]
| (main:) Problem starting service jboss.messaging:service=PostOffice
| java.lang.NullPointerException
| at
| org.jboss.messaging.core.impl.postoffice.GroupMember.start(GroupMember.java:160)
| at
| org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.start(MessagingPostOffice.java:347)
| at
| org.jboss.messaging.core.jmx.MessagingPostOfficeService.startService(MessagingPostOfficeService.java:427)
| at
| org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
| at
| org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
| sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at
| org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at
| org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
| at $Proxy0.start(Unknown Source)
| at org.jboss.system.ServiceController.start(ServiceController.java:417)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy4.start(Unknown Source)
| at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
| sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at
| org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy43.start(Unknown Source)
| at org.jboss.deployment.XSLSubDeployer.start(XSLSubDeployer.java:197)
| at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
| at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at
| org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy9.deploy(Unknown Source)
| at
| org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
| at
| org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
| at
| org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
| at
| org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
| at
| org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
| at
| org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
| at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at
| org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
| at $Proxy0.start(Unknown Source)
| at org.jboss.system.ServiceController.start(ServiceController.java:417)
| at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy4.start(Unknown Source)
| at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
| at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
| sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at
| org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy5.deploy(Unknown Source)
| at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
| at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
| at org.jboss.Main.boot(Main.java:200)
| at org.jboss.Main$1.run(Main.java:508)
| at java.lang.Thread.run(Thread.java:595)
JGroups stack used for controll channel:
<stack name="udp-sync"
| description="IP multicast based stack, without flow control and without message bundling. This should be used
| instead of udp if (1) synchronous calls are used and (2) the message volume (rate and size)
| is not that large. Don't use this configuration if you send messages at a high sustained rate, or you might
| run out of memory">
| <config>
| <UDP
| mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
| mcast_port="${jgroups.udp.mcast_port:45599}"
| tos="8"
| ucast_recv_buf_size="20000000"
| ucast_send_buf_size="640000"
| mcast_recv_buf_size="25000000"
| mcast_send_buf_size="640000"
| loopback="false"
| discard_incompatible_packets="true"
| max_bundle_size="64000"
| max_bundle_timeout="30"
| use_incoming_packet_handler="true"
| ip_ttl="${jgroups.udp.ip_ttl:2}"
| enable_bundling="true"
|
| use_concurrent_stack="true"
|
| thread_pool.enabled="true"
| thread_pool.min_threads="1"
| thread_pool.max_threads="25"
| thread_pool.keep_alive_time="5000"
| thread_pool.queue_enabled="false"
| thread_pool.queue_max_size="100"
| thread_pool.rejection_policy="Run"
|
| oob_thread_pool.enabled="true"
| oob_thread_pool.min_threads="1"
| oob_thread_pool.max_threads="8"
| oob_thread_pool.keep_alive_time="5000"
| oob_thread_pool.queue_enabled="false"
| oob_thread_pool.queue_max_size="100"
| oob_thread_pool.rejection_policy="Run"/>
| <PING timeout="2000"
| num_initial_members="3"/>
| <MERGE2 max_interval="100000"
| min_interval="20000"/>
| <FD_SOCK/>
| <FD timeout="10000" max_tries="5" shun="true"/>
| <VERIFY_SUSPECT timeout="1500" />
| <pbcast.NAKACK max_xmit_size="60000"
| use_mcast_xmit="false" gc_lag="0"
| retransmit_timeout="300,600,1200,2400,4800"
| discard_delivered_msgs="false"/>
| <UNICAST timeout="300,600,1200,2400,3600"/>
| <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
| max_bytes="400000"/>
| <pbcast.GMS print_local_addr="true" join_timeout="3000"
| join_retry_timeout="2000" shun="false"
| view_bundling="true"/>
| <FRAG2 frag_size="60000" />
| <!--
| <pbcast.STREAMING_STATE_TRANSFER/>
| -->
| <pbcast.STATE_TRANSFER/>
| <pbcast.FLUSH timeout="0"/>
| </config>
| </stack>
|
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4121295#4121295
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4121295
More information about the jboss-user
mailing list