[jboss-jira] [JBoss JIRA] (WFLY-3727) Race condition during channel create vs start causing NPE on channel startup

[ https://issues.jboss.org/browse/WFLY-3727?page=com.atlassian.jira.plugin.... ] Paul Ferraro updated WFLY-3727: ------------------------------- Description: My hunch is that this is due to the fact that WildFly creates a channel in one thread but starts it in another. Specifically, Protocol.up_prot is set by ChannelService.start() (via new JChannel(...)) but is accessed by GlobalComponentRegistryService.start() (via Channel.connect(...)) - which happen in different threads. To prevents NPEs like the one below, we need to ensure that any variables set during channel creation are volatile. {noformat} ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 30) MSC000001: Failed to start service jboss.infinispan.ejb.global-component-registry: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.global-component-registry: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final] Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:241) at org.jboss.as.clustering.infinispan.subsystem.GlobalComponentRegistryService.start(GlobalComponentRegistryService.java:33) at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) ... 4 more Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:219) ... 6 more Caused by: org.infinispan.commons.CacheException: Unable to start JGroups Channel at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:198) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:187) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183) ... 11 more Caused by: java.lang.NullPointerException at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:391) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) at org.jgroups.protocols.FD.up(FD.java:255) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301) at org.jgroups.protocols.MERGE2.up(MERGE2.java:209) at org.jgroups.protocols.Discovery.up(Discovery.java:522) at org.jgroups.protocols.MPING.up(MPING.java:181) at org.jgroups.protocols.TP.fetchLocalAddresses(TP.java:2001) at org.jgroups.protocols.TP.start(TP.java:1131) at org.jgroups.protocols.TCP.start(TCP.java:82) at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:938) at org.jgroups.JChannel.startStack(JChannel.java:864) at org.jgroups.JChannel._preConnect(JChannel.java:527) at org.jgroups.JChannel.connect(JChannel.java:284) at org.jgroups.JChannel.connect(JChannel.java:275) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:196) ... 17 more {noformat} was: My hunch is that this is due to the fact that WildFly creates a channel in one thread but starts it in another. Specifically, Protocol.up_prot is set by ChannelService.start() but is accessed by GlobalComponentRegistryService.start() - which happen in different threads. To prevents NPEs like the one below, we need to ensure that any variables set during channel creation are volatile. {noformat} ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 30) MSC000001: Failed to start service jboss.infinispan.ejb.global-component-registry: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.global-component-registry: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final] Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:241) at org.jboss.as.clustering.infinispan.subsystem.GlobalComponentRegistryService.start(GlobalComponentRegistryService.java:33) at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) ... 4 more Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:219) ... 6 more Caused by: org.infinispan.commons.CacheException: Unable to start JGroups Channel at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:198) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:187) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183) ... 11 more Caused by: java.lang.NullPointerException at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:391) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) at org.jgroups.protocols.FD.up(FD.java:255) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301) at org.jgroups.protocols.MERGE2.up(MERGE2.java:209) at org.jgroups.protocols.Discovery.up(Discovery.java:522) at org.jgroups.protocols.MPING.up(MPING.java:181) at org.jgroups.protocols.TP.fetchLocalAddresses(TP.java:2001) at org.jgroups.protocols.TP.start(TP.java:1131) at org.jgroups.protocols.TCP.start(TCP.java:82) at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:938) at org.jgroups.JChannel.startStack(JChannel.java:864) at org.jgroups.JChannel._preConnect(JChannel.java:527) at org.jgroups.JChannel.connect(JChannel.java:284) at org.jgroups.JChannel.connect(JChannel.java:275) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:196) ... 17 more {noformat} -- This message was sent by Atlassian JIRA (v6.2.6#6264)