[infinispan-issues] [JBoss JIRA] (ISPN-6099) ConcurrentJoinTest random failures

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154524#comment-13154524 ] 

Dan Berindei commented on ISPN-6099:
------------------------------------

The [{{socket()}} man page|http://linux.die.net/man/7/socket] says about {{SO_REUSEADDR}} "For AF_INET sockets this means that a socket may bind, except when there is an active listening socket bound to the address." So 2 sockets can bind to the same port, as long as only one of them calls {{listen()}} and becomes an _active listening socket_ (i.e. a _server socket_).

In Java, {{ServerSocket}} only has a {{bind()}} method, which suggests that it atomically binds to the port and listens to it. But in fact this is not atomic, and it's entirely possible to fail in the {{listen()}} system call. When that happens, {{ServerSocket.bind()}} throws a {{SocketException}} (as opposed to a {{BindException}}, if the failure happens in the {{bind()}} system call). The {{ServerSocket}} then becomes invalid, because its corresponding OS socket if bound to a port, but it can't accept connections, and calling {{ServerSocket.bind()}} again will fail - this time with a {{BindException}}.



> ConcurrentJoinTest random failures
> ----------------------------------
>
>                 Key: ISPN-6099
>                 URL: https://issues.jboss.org/browse/ISPN-6099
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Test Suite - Core
>    Affects Versions: 8.1.0.Final
>         Environment: java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>            Reporter: Dan Berindei
>            Assignee: Dan Berindei
>             Fix For: 8.2.0.Beta1
>
>
> Since the switch to {{TCP_NIO2}} in the test suite, I've been seeing random failures in {{ConcurrentJoinTest}} and other tests that attempt to start multiple channels in parallel (e.g. {{StateTransferFunctionalTest}} and its subclasses).
> Normally JGroups only reports a {{java.net.BindException: No available port to bind to in range [8000 .. 8099]}}, but I have modified {{org.jgroups.util.Util.createServerSocket()}} to report the cause exception and I got this:
> {noformat}
> java.net.BindException: No available port to bind to in range [8000 .. 8099]
>         at org.jgroups.util.Util.createServerSocketChannel(Util.java:3077) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
>         at org.jgroups.blocks.cs.NioServer.<init>(NioServer.java:86) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
>         at org.jgroups.protocols.TCP_NIO2.start(TCP_NIO2.java:97) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
>         at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:966) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
>         at org.jgroups.JChannel.startStack(JChannel.java:890) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
>         at org.jgroups.JChannel._preConnect(JChannel.java:553) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
>         at org.jgroups.JChannel.connect(JChannel.java:288) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
>         at org.jgroups.JChannel.connect(JChannel.java:279) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
>         at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:199) ~[classes/:?]
>         at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:190) ~[classes/:?]
>         at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) ~[?:?]
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
>         at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
>         at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) ~[infinispan-commons-8.2.0-SNAPSHOT.jar:8.2.0-SNAPSHOT]
>         at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:870) ~[classes/:?]
>         at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639) ~[classes/:?]
>         at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:628) ~[classes/:?]
>         at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:531) ~[classes/:?]
>         at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:229) ~[classes/:?]
>         ... 11 more
> Caused by: java.net.SocketException: Invalid argument
>         at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_60]
>         at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_60]
>         at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_60]
>         at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:1.8.0_60]
>         at java.nio.channels.ServerSocketChannel.bind(ServerSocketChannel.java:157) ~[?:1.8.0_60]
>         at org.jgroups.util.Util.createServerSocketChannel(Util.java:3072) ~[jgroups-3.6.7.Final.jar:3.6.7.Final]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)