8:39 a.m.
When we start up our JBoss application we sometimes get the exception shown below. It
occurs on roughly 1/4 of startups. We are running JBossAS 4.2.0.GA with JBoss Messaging
1.4.0.SP3 (also saw problem with SP2). We are running with clustered destinations (but
with just one cluster node at present). We use JGroups 2.5.1 with the multiplexor using
the udp-sync channel config (shown below) for the control channel.
Debugging, the following seems to happen at application server startup:
- Main thread calls GroupMember.start() which connects to the controlChannel
- Multiplexer thread calls back to the ControlMembershipListener viewAccepted() which
creates the latch object
- Main thread continues to call latch.countDown(). There is clearly a race condition here
which causes latch to be null sometimes. Bang!
Note:
- problem does not occur if dont use multiplexor in JBoss messaging config (i.e. use
configuration bundled with JBM): i.e. callback to viewAccepted() is on the main thread.
- but it does occur if use multiplexor with the udp-sync channel config replaced by that
in the configuration bundled with JBM. This differs from that in udp-sync (below) in that
in the former use_flush is set for STATE_TRANSFER and GMS, and in udp-sync
use_concurrent_stack=true with thread_pool and oob.thread_pool.enabled both =true. So
these differences do not account for the problem.
Setting the system property jgroups.mux.enabled=false makes it work again (with the
multiplexor-style configuration) and is our solution for the time being.
Any thoughts?
2008-01-17 19:02:33,118 37694 WARN [org.jboss.system.ServiceController]
| (main:) Problem starting service jboss.messaging:service=PostOffice
| java.lang.NullPointerException
| at
| org.jboss.messaging.core.impl.postoffice.GroupMember.start(GroupMember.java:160)
| at
|
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.start(MessagingPostOffice.java:347)
| at
|
org.jboss.messaging.core.jmx.MessagingPostOfficeService.startService(MessagingPostOfficeService.java:427)
| at
| org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
| at
| org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
| sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at
| org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
|
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at
| org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
| at $Proxy0.start(Unknown Source)
| at org.jboss.system.ServiceController.start(ServiceController.java:417)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at org.jboss.system.ServiceController.start(ServiceController.java:435)
| at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy4.start(Unknown Source)
| at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
| sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at
| org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
|
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy43.start(Unknown Source)
| at org.jboss.deployment.XSLSubDeployer.start(XSLSubDeployer.java:197)
| at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
| at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at
| org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
|
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy9.deploy(Unknown Source)
| at
|
org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
| at
| org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
| at
|
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
| at
|
org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
| at
| org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
| at
|
org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
| at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at
| org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
| at $Proxy0.start(Unknown Source)
| at org.jboss.system.ServiceController.start(ServiceController.java:417)
| at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy4.start(Unknown Source)
| at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
| at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
| at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
| sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at
| sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at
| org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
| at
| org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
|
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at
| org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
| at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
| at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
| at $Proxy5.deploy(Unknown Source)
| at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
| at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
| at org.jboss.Main.boot(Main.java:200)
| at org.jboss.Main$1.run(Main.java:508)
| at java.lang.Thread.run(Thread.java:595)
JGroups stack used for controll channel:
<stack name="udp-sync"
| description="IP multicast based stack, without flow control and
without message bundling. This should be used
| instead of udp if (1) synchronous calls are used and (2) the message volume
(rate and size)
| is not that large. Don't use this configuration if you send messages
at a high sustained rate, or you might
| run out of memory">
| <config>
| <UDP
| mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
| mcast_port="${jgroups.udp.mcast_port:45599}"
| tos="8"
| ucast_recv_buf_size="20000000"
| ucast_send_buf_size="640000"
| mcast_recv_buf_size="25000000"
| mcast_send_buf_size="640000"
| loopback="false"
| discard_incompatible_packets="true"
| max_bundle_size="64000"
| max_bundle_timeout="30"
| use_incoming_packet_handler="true"
| ip_ttl="${jgroups.udp.ip_ttl:2}"
| enable_bundling="true"
|
| use_concurrent_stack="true"
|
| thread_pool.enabled="true"
| thread_pool.min_threads="1"
| thread_pool.max_threads="25"
| thread_pool.keep_alive_time="5000"
| thread_pool.queue_enabled="false"
| thread_pool.queue_max_size="100"
| thread_pool.rejection_policy="Run"
|
| oob_thread_pool.enabled="true"
| oob_thread_pool.min_threads="1"
| oob_thread_pool.max_threads="8"
| oob_thread_pool.keep_alive_time="5000"
| oob_thread_pool.queue_enabled="false"
| oob_thread_pool.queue_max_size="100"
| oob_thread_pool.rejection_policy="Run"/>
| <PING timeout="2000"
| num_initial_members="3"/>
| <MERGE2 max_interval="100000"
| min_interval="20000"/>
| <FD_SOCK/>
| <FD timeout="10000" max_tries="5"
shun="true"/>
| <VERIFY_SUSPECT timeout="1500" />
| <pbcast.NAKACK max_xmit_size="60000"
| use_mcast_xmit="false" gc_lag="0"
| retransmit_timeout="300,600,1200,2400,4800"
| discard_delivered_msgs="false"/>
| <UNICAST timeout="300,600,1200,2400,3600"/>
| <pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000"
| max_bytes="400000"/>
| <pbcast.GMS print_local_addr="true"
join_timeout="3000"
| join_retry_timeout="2000" shun="false"
| view_bundling="true"/>
| <FRAG2 frag_size="60000" />
| <!--
| <pbcast.STREAMING_STATE_TRANSFER/>
| -->
| <pbcast.STATE_TRANSFER/>
| <pbcast.FLUSH timeout="0"/>
| </config>
| </stack>
|
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4121295#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...