FD_SOCK port conflicts with concurrently connected channels ----------------------------------------------------------- Key: JGRP-1314 URL: https://issues.jboss.org/browse/JGRP-1314 Project: JGroups Issue Type: Bug Affects Versions: 2.12.1 Reporter: Paul Ferraro Assignee: Bela Ban Because of the high level of concurrency in AS7, I'm seeing occasional socket exceptions in FD_SOCK when trying to start multiple channels using a shared transport. It appears that there are some thread safety issues with the port increment logic within FD_SOCK. Until JGRP-790 is complete, measures need to be taken to ensure that multiple concurrently connected channels do not select the same port on which to open a server socket. Here's a sample stack trace: 10:01:26,716 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC00001: Failed to start service jboss.infinispan.cluster: org.jboss.msc.service.StartException in service jboss.infinispan.cluster: Failed to start service at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1696) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [:1.6.0_22] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [:1.6.0_22] at java.lang.Thread.run(Thread.java:679) [:1.6.0_22] Caused by: org.infinispan.CacheException: Unable to invoke method public abstract void org.infinispan.remoting.transport.Transport.start() on object at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:174) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:687) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:589) at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:137) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:137) at org.infinispan.CacheDelegate.start(CacheDelegate.java:323) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:439) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:77) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:66) at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.start(EmbeddedCacheManagerService.java:171) at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1675) ... 3 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_22] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [:1.6.0_22] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_22] at java.lang.reflect.Method.invoke(Method.java:616) [:1.6.0_22] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171) ... 15 more Caused by: java.lang.IllegalArgumentException: failed to start server socket at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:295) at org.jgroups.protocols.FD.down(FD.java:310) at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:70) at org.jgroups.protocols.BARRIER.down(BARRIER.java:90) at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:634) at org.jgroups.protocols.UNICAST.down(UNICAST.java:443) at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:330) at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:186) at org.jgroups.protocols.pbcast.GMS.down(GMS.java:926) at org.jgroups.protocols.FlowControl.down(FlowControl.java:351) at org.jgroups.protocols.FlowControl.down(FlowControl.java:351) at org.jgroups.protocols.FRAG2.down(FRAG2.java:155) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.down(STREAMING_STATE_TRANSFER.java:324) at org.jgroups.protocols.pbcast.FLUSH.handleConnect(FLUSH.java:302) at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:263) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1081) at org.jgroups.JChannel.downcall(JChannel.java:1680) at org.jgroups.JChannel.connect(JChannel.java:425) at org.jgroups.JChannel.connect(JChannel.java:388) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:171) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:155) ... 20 more Caused by: java.net.SocketException: Address already in use at java.net.PlainSocketImpl.socketListen(Native Method) [:1.6.0_22] at java.net.AbstractPlainSocketImpl.listen(AbstractPlainSocketImpl.java:365) [:1.6.0_22] at java.net.ServerSocket.bind(ServerSocket.java:337) [:1.6.0_22] at java.net.ServerSocket.<init>(ServerSocket.java:202) [:1.6.0_22] at org.jgroups.util.DefaultSocketFactory.createServerSocket(DefaultSocketFactory.java:49) at org.jgroups.util.Util.createServerSocket(Util.java:3046) at org.jgroups.protocols.FD_SOCK.startServerSocket(FD_SOCK.java:561) at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:292) ... 40 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira