[jboss-jira] [JBoss JIRA] Created: (JGRP-1314) FD_SOCK port conflicts with concurrently connected channels

[Paul Ferraro (JIRA)]

FD_SOCK port conflicts with concurrently connected channels
-----------------------------------------------------------

                 Key: JGRP-1314
                 URL: https://issues.jboss.org/browse/JGRP-1314
             Project: JGroups
          Issue Type: Bug
    Affects Versions: 2.12.1
            Reporter: Paul Ferraro
            Assignee: Bela Ban


Because of the high level of concurrency in AS7, I'm seeing occasional socket exceptions in FD_SOCK when trying to start multiple channels using a shared transport.  It appears that there are some thread safety issues with the port increment logic within FD_SOCK.  
Until JGRP-790 is complete, measures need to be taken to ensure that multiple concurrently connected channels do not select the same port on which to open a server socket.

Here's a sample stack trace:

10:01:26,716 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC00001: Failed to start service jboss.infinispan.cluster: org.jboss.msc.service.StartException in service jboss.infinispan.cluster: Failed to start service
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1696)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [:1.6.0_22]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [:1.6.0_22]
	at java.lang.Thread.run(Thread.java:679) [:1.6.0_22]
Caused by: org.infinispan.CacheException: Unable to invoke method public abstract void org.infinispan.remoting.transport.Transport.start() on object 
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:174)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:687)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:589)
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:137)
	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:137)
	at org.infinispan.CacheDelegate.start(CacheDelegate.java:323)
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:439)
	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:77)
	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:66)
	at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.start(EmbeddedCacheManagerService.java:171)
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1675)
	... 3 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_22]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [:1.6.0_22]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_22]
	at java.lang.reflect.Method.invoke(Method.java:616) [:1.6.0_22]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
	... 15 more
Caused by: java.lang.IllegalArgumentException: failed to start server socket
	at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:295)
	at org.jgroups.protocols.FD.down(FD.java:310)
	at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:70)
	at org.jgroups.protocols.BARRIER.down(BARRIER.java:90)
	at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:634)
	at org.jgroups.protocols.UNICAST.down(UNICAST.java:443)
	at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:330)
	at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:186)
	at org.jgroups.protocols.pbcast.GMS.down(GMS.java:926)
	at org.jgroups.protocols.FlowControl.down(FlowControl.java:351)
	at org.jgroups.protocols.FlowControl.down(FlowControl.java:351)
	at org.jgroups.protocols.FRAG2.down(FRAG2.java:155)
	at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.down(STREAMING_STATE_TRANSFER.java:324)
	at org.jgroups.protocols.pbcast.FLUSH.handleConnect(FLUSH.java:302)
	at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:263)
	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1081)
	at org.jgroups.JChannel.downcall(JChannel.java:1680)
	at org.jgroups.JChannel.connect(JChannel.java:425)
	at org.jgroups.JChannel.connect(JChannel.java:388)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:171)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:155)
	... 20 more

Caused by: java.net.SocketException: Address already in use
	at java.net.PlainSocketImpl.socketListen(Native Method) [:1.6.0_22]
	at java.net.AbstractPlainSocketImpl.listen(AbstractPlainSocketImpl.java:365) [:1.6.0_22]
	at java.net.ServerSocket.bind(ServerSocket.java:337) [:1.6.0_22]
	at java.net.ServerSocket.<init>(ServerSocket.java:202) [:1.6.0_22]
	at org.jgroups.util.DefaultSocketFactory.createServerSocket(DefaultSocketFactory.java:49)
	at org.jgroups.util.Util.createServerSocket(Util.java:3046)
	at org.jgroups.protocols.FD_SOCK.startServerSocket(FD_SOCK.java:561)
	at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:292)
	... 40 more


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira