Investigate Netty's "Too many open files" exception in testing of infinispan-compatibility-mode-it -------------------------------------------------------------------------------------------------- Key: ISPN-7517 URL: https://issues.jboss.org/browse/ISPN-7517 Project: Infinispan Issue Type: Task Components: Test Suite - Server Reporter: Sanne Grinovero Assignee: Dan Berindei Fix For: 9.0.0.Final See also discussion on ML: - http://lists.jboss.org/pipermail/infinispan-dev/2017-February/017232.html Running: {noformat} mci -rf :infinispan-compatibility-mode-it {noformat} {noformat} Failed tests: ByteArrayKeyDistEmbeddedHotRodTest.setup:27 » IllegalState failed to create a ... ByteArrayKeyReplEmbeddedHotRodTest.setup:87 » IllegalState failed to create a ... ByteArrayValueDistEmbeddedHotRodTest.setup:27 » IllegalState failed to create ... ByteArrayValueReplEmbeddedHotRodTest.setup:87 » IllegalState failed to create ... DistEmbeddedRestHotRodTest.setup:25 » IllegalState failed to create a child ev... DistL1EmbeddedHotRodTest.setup:30 » IllegalState failed to create a child even... DistMemcachedEmbeddedTest.setup:39 » IllegalState failed to create a child eve... org.infinispan.it.compatibility.EmbeddedHotRodCacheListenerTest.setup(org.infinispan.it.compatibility.EmbeddedHotRodCacheListenerTest) Run 1: EmbeddedHotRodCacheListenerTest.setup:36 » IllegalState failed to create a chi... Run 2: PASS Run 3: PASS EmbeddedHotRodTest.testEventFilteringStatic:274->lambda$testEventFilteringStatic$4:276 » Transport EmbeddedMemcachedCacheListenerTest.setup:39 » IllegalState failed to create a ... EmbeddedRestMemcachedHotRodTest.setup:50 » IllegalState failed to create a chi... ReplEmbeddedRestHotRodTest.setup:38 » IllegalState failed to create a child ev... Tests run: 101, Failures: 12, Errors: 0, Skipped: 47 {noformat} Errors are all about a similar stacktrace. {noformat} java.lang.IllegalStateException: failed to create a child event loop at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:51) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:87) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:82) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:63) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:51) at org.jboss.resteasy.plugins.server.netty.NettyJaxrsServer.start(NettyJaxrsServer.java:239) at org.infinispan.rest.NettyRestServer.start(NettyRestServer.java:81) at org.infinispan.it.compatibility.CompatibilityCacheFactory.createRestCache(CompatibilityCacheFactory.java:199) at org.infinispan.it.compatibility.CompatibilityCacheFactory.createRestMemcachedCaches(CompatibilityCacheFactory.java:137) at org.infinispan.it.compatibility.CompatibilityCacheFactory.setup(CompatibilityCacheFactory.java:123) at org.infinispan.it.compatibility.DistMemcachedEmbeddedTest.setup(DistMemcachedEmbeddedTest.java:39) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84) at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138) at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107) at org.testng.TestRunner.privateRun(TestRunner.java:767) at org.testng.TestRunner.run(TestRunner.java:617) at org.testng.SuiteRunner.runTest(SuiteRunner.java:348) at org.testng.SuiteRunner.access$000(SuiteRunner.java:38) at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382) at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: io.netty.channel.ChannelException: failed to open a new selector at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:157) at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:148) at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:126) at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ... 32 more Caused by: java.io.IOException: Too many open files at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:130) at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:69) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:155) ... 36 more {noformat} N.B. The failure is quite consistent on my workstation, almost always results in the same tests failing. Yet occasionally they all pass.