[jboss-jira] [JBoss JIRA] (WFLY-3727) Race condition during channel create vs start causing NPE on channel startup

Paul Ferraro (JIRA) issues at jboss.org
Mon Aug 11 14:36:29 EDT 2014 

     [ https://issues.jboss.org/browse/WFLY-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Ferraro updated WFLY-3727:
-------------------------------

    Description: 
My hunch is that this is due to the fact that WildFly creates a channel in one thread but starts it in another.  Specifically, Protocol.up_prot is set by ChannelService.start() but is accessed by GlobalComponentRegistryService.start() - which happen in different threads.  To prevents NPEs like the one below, we need to ensure that any variables set during channel creation are volatile.
{noformat}
ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 30) MSC000001: Failed to start service jboss.infinispan.ejb.global-component-registry: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.global-component-registry: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
    at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
    at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
    at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:241)
    at org.jboss.as.clustering.infinispan.subsystem.GlobalComponentRegistryService.start(GlobalComponentRegistryService.java:33)
    at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86)
    ... 4 more
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
    at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
    at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
    at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:219)
    ... 6 more
Caused by: org.infinispan.commons.CacheException: Unable to start JGroups Channel
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:198)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:187)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
    ... 11 more
Caused by: java.lang.NullPointerException
    at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
    at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:391)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
    at org.jgroups.protocols.FD.up(FD.java:255)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
    at org.jgroups.protocols.Discovery.up(Discovery.java:522)
    at org.jgroups.protocols.MPING.up(MPING.java:181)
    at org.jgroups.protocols.TP.fetchLocalAddresses(TP.java:2001)
    at org.jgroups.protocols.TP.start(TP.java:1131)
    at org.jgroups.protocols.TCP.start(TCP.java:82)
    at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:938)
    at org.jgroups.JChannel.startStack(JChannel.java:864)
    at org.jgroups.JChannel._preConnect(JChannel.java:527)
    at org.jgroups.JChannel.connect(JChannel.java:284)
    at org.jgroups.JChannel.connect(JChannel.java:275)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:196)
    ... 17 more
{noformat}

  was:
My hunch is that this is due to the fact that WildFly creates a channel in one thread but starts it in another.  To prevents NPEs like the one below, we need to ensure that any variable set during channel creation are volatile.
{noformat}
ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 30) MSC000001: Failed to start service jboss.infinispan.ejb.global-component-registry: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.global-component-registry: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
    at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
    at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
    at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:241)
    at org.jboss.as.clustering.infinispan.subsystem.GlobalComponentRegistryService.start(GlobalComponentRegistryService.java:33)
    at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86)
    ... 4 more
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
    at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
    at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
    at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:219)
    ... 6 more
Caused by: org.infinispan.commons.CacheException: Unable to start JGroups Channel
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:198)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:187)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
    ... 11 more
Caused by: java.lang.NullPointerException
    at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
    at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:391)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
    at org.jgroups.protocols.FD.up(FD.java:255)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
    at org.jgroups.protocols.Discovery.up(Discovery.java:522)
    at org.jgroups.protocols.MPING.up(MPING.java:181)
    at org.jgroups.protocols.TP.fetchLocalAddresses(TP.java:2001)
    at org.jgroups.protocols.TP.start(TP.java:1131)
    at org.jgroups.protocols.TCP.start(TCP.java:82)
    at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:938)
    at org.jgroups.JChannel.startStack(JChannel.java:864)
    at org.jgroups.JChannel._preConnect(JChannel.java:527)
    at org.jgroups.JChannel.connect(JChannel.java:284)
    at org.jgroups.JChannel.connect(JChannel.java:275)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:196)
    ... 17 more
{noformat}



> Race condition during channel create vs start causing NPE on channel startup
> ----------------------------------------------------------------------------
>
>                 Key: WFLY-3727
>                 URL: https://issues.jboss.org/browse/WFLY-3727
>             Project: WildFly
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Clustering
>    Affects Versions: 8.1.0.Final
>            Reporter: Paul Ferraro
>            Assignee: Paul Ferraro
>             Fix For: 9.0.0.Beta1
>
>
> My hunch is that this is due to the fact that WildFly creates a channel in one thread but starts it in another.  Specifically, Protocol.up_prot is set by ChannelService.start() but is accessed by GlobalComponentRegistryService.start() - which happen in different threads.  To prevents NPEs like the one below, we need to ensure that any variables set during channel creation are volatile.
> {noformat}
> ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 30) MSC000001: Failed to start service jboss.infinispan.ejb.global-component-registry: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.global-component-registry: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
>     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
>     at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
>     at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
> Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
>     at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:241)
>     at org.jboss.as.clustering.infinispan.subsystem.GlobalComponentRegistryService.start(GlobalComponentRegistryService.java:33)
>     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86)
>     ... 4 more
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport
>     at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
>     at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
>     at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
>     at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
>     at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
>     at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:219)
>     ... 6 more
> Caused by: org.infinispan.commons.CacheException: Unable to start JGroups Channel
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:198)
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:187)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51]
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51]
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
>     at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
>     at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
>     ... 11 more
> Caused by: java.lang.NullPointerException
>     at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
>     at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
>     at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:391)
>     at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
>     at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
>     at org.jgroups.protocols.FD.up(FD.java:255)
>     at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
>     at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
>     at org.jgroups.protocols.Discovery.up(Discovery.java:522)
>     at org.jgroups.protocols.MPING.up(MPING.java:181)
>     at org.jgroups.protocols.TP.fetchLocalAddresses(TP.java:2001)
>     at org.jgroups.protocols.TP.start(TP.java:1131)
>     at org.jgroups.protocols.TCP.start(TCP.java:82)
>     at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:938)
>     at org.jgroups.JChannel.startStack(JChannel.java:864)
>     at org.jgroups.JChannel._preConnect(JChannel.java:527)
>     at org.jgroups.JChannel.connect(JChannel.java:284)
>     at org.jgroups.JChannel.connect(JChannel.java:275)
>     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:196)
>     ... 17 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2.6#6264)

More information about the jboss-jira mailing list