[JBoss JIRA] (ISPN-7517) Investigate Netty's "Too many open files" exception in testing of infinispan-compatibility-mode-it
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7517?page=com.atlassian.jira.plugin.... ]
Dan Berindei edited comment on ISPN-7517 at 2/27/17 9:28 AM:
-------------------------------------------------------------
Looks like there is no leak after all: I added a 5 seconds delay before logging the leaked threads in {{TestNGTestListener.onFinish(org.testng.ISuite)}}, and now it only reports a few {{commonPool}} worker threads, no HotRod/Memcached worker threads.
The problem seems to be in {{NettyTransport.stop()}}, which calls {{EventLoopGroup.shutdownGracefully()}}, but then ignores the termination future.
Of course, the {{compatibility-mode-it}} could be further improved by starting less servers, either by using a single set of servers for the whole test suite, or by starting only the server that the test is actually using. The servers could also be configured to start a lot less threads than the production defaults.
was (Author: dan.berindei):
Looks like there is no leak after all: I added a 5 seconds delay before logging the leaked threads in {{TestNGTestListener.onFinish(org.testng.ISuite)}}, and now it only reports a few {{commonPool}} worker threads, no HotRod/Memcached worker threads.
The problem seems to be in {{AbstractProtocolServer.stop()}}, which calls {{EventLoopGroup.shutdownGracefully()}}, but then ignores the termination future.
Of course, the {{compatibility-mode-it}} could be further improved by starting less servers, either by using a single set of servers for the whole test suite, or by starting only the server that the test is actually using. The servers could also be configured to start a lot less threads than the production defaults.
> Investigate Netty's "Too many open files" exception in testing of infinispan-compatibility-mode-it
> --------------------------------------------------------------------------------------------------
>
> Key: ISPN-7517
> URL: https://issues.jboss.org/browse/ISPN-7517
> Project: Infinispan
> Issue Type: Task
> Components: Test Suite - Server
> Reporter: Sanne Grinovero
> Fix For: 9.0.0.Final
>
>
> See also discussion on ML:
> - http://lists.jboss.org/pipermail/infinispan-dev/2017-February/017232.html
> Running:
> {noformat}
> mci -rf :infinispan-compatibility-mode-it
> {noformat}
> {noformat}
> Failed tests:
> ByteArrayKeyDistEmbeddedHotRodTest.setup:27 » IllegalState failed to create a ...
> ByteArrayKeyReplEmbeddedHotRodTest.setup:87 » IllegalState failed to create a ...
> ByteArrayValueDistEmbeddedHotRodTest.setup:27 » IllegalState failed to create ...
> ByteArrayValueReplEmbeddedHotRodTest.setup:87 » IllegalState failed to create ...
> DistEmbeddedRestHotRodTest.setup:25 » IllegalState failed to create a child ev...
> DistL1EmbeddedHotRodTest.setup:30 » IllegalState failed to create a child even...
> DistMemcachedEmbeddedTest.setup:39 » IllegalState failed to create a child eve...
> org.infinispan.it.compatibility.EmbeddedHotRodCacheListenerTest.setup(org.infinispan.it.compatibility.EmbeddedHotRodCacheListenerTest)
> Run 1: EmbeddedHotRodCacheListenerTest.setup:36 » IllegalState failed to create a chi...
> Run 2: PASS
> Run 3: PASS
> EmbeddedHotRodTest.testEventFilteringStatic:274->lambda$testEventFilteringStatic$4:276 » Transport
> EmbeddedMemcachedCacheListenerTest.setup:39 » IllegalState failed to create a ...
> EmbeddedRestMemcachedHotRodTest.setup:50 » IllegalState failed to create a chi...
> ReplEmbeddedRestHotRodTest.setup:38 » IllegalState failed to create a child ev...
> Tests run: 101, Failures: 12, Errors: 0, Skipped: 47
> {noformat}
> Errors are all about a similar stacktrace.
> {noformat}
> java.lang.IllegalStateException: failed to create a child event loop
> at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
> at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
> at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:51)
> at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:87)
> at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:82)
> at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:63)
> at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:51)
> at org.jboss.resteasy.plugins.server.netty.NettyJaxrsServer.start(NettyJaxrsServer.java:239)
> at org.infinispan.rest.NettyRestServer.start(NettyRestServer.java:81)
> at org.infinispan.it.compatibility.CompatibilityCacheFactory.createRestCache(CompatibilityCacheFactory.java:199)
> at org.infinispan.it.compatibility.CompatibilityCacheFactory.createRestMemcachedCaches(CompatibilityCacheFactory.java:137)
> at org.infinispan.it.compatibility.CompatibilityCacheFactory.setup(CompatibilityCacheFactory.java:123)
> at org.infinispan.it.compatibility.DistMemcachedEmbeddedTest.setup(DistMemcachedEmbeddedTest.java:39)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)
> at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: io.netty.channel.ChannelException: failed to open a new selector
> at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:157)
> at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:148)
> at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:126)
> at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36)
> at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
> ... 32 more
> Caused by: java.io.IOException: Too many open files
> at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
> at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:130)
> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:69)
> at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:155)
> ... 36 more
> {noformat}
> N.B.
> The failure is quite consistent on my workstation, almost always results in the same tests failing. Yet occasionally they all pass.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 2 months
[JBoss JIRA] (ISPN-5665) Query should not rely on the results of return values of write commands
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5665?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5665:
----------------------------------
Fix Version/s: (was: 9.0.0.Final)
> Query should not rely on the results of return values of write commands
> -----------------------------------------------------------------------
>
> Key: ISPN-5665
> URL: https://issues.jboss.org/browse/ISPN-5665
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 8.0.0.Beta2
> Reporter: Dan Berindei
> Assignee: Gustavo Fernandes
>
> The query interceptor relies on the return value of the write commands to know the previous value of the modified entries. This is not correct, because some write commands do not return the previous value, e.g. {{remove(key, value)}}, {{replace(key, oldValue, newValue)}}, and {{putAll(map)}}.
> The query interceptor should instead look up the previous values in the invocation context, and also force the loading of old values in the invocation context if the command doesn't do it explicitly (e.g. {{putAll(map)}}, or {{put(k, v)}} with the {{IGNORE_RETURN_VALUES}} flag).
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 2 months
[JBoss JIRA] (ISPN-7489) org.jgroups.protocols.TCP emits errors when node leaves the cluster
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7489?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-7489:
--------------------------------------
Status: Open (was: New)
> org.jgroups.protocols.TCP emits errors when node leaves the cluster
> -------------------------------------------------------------------
>
> Key: ISPN-7489
> URL: https://issues.jboss.org/browse/ISPN-7489
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud Integrations, Core
> Affects Versions: 9.0.0.CR1
> Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}}
> * Custom Infinispan Server build (based on [these instructions|https://github.com/slaskawi/infinispan-1/tree/custom_image]). SHA1 {{2b0731b21649a88a75ed71d21b9cc06ba365e947}}
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Priority: Critical
>
> When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems.
> One of the is this:
> {noformat}
> [transactions-repository-1-04x09] 18:09:12,193 ERROR [org.jgroups.protocols.TCP] (jgroups-30,transactions-repository-1-04x09) JGRP000029: transactions-repository-1-04x09: failed sending message to transactions-repository-1-4z05w (71 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=5262, TP: [cluster_name=cluster]
> [transactions-repository-1-1f8dx] 18:09:12,310 ERROR [org.jgroups.protocols.TCP] (jgroups-16,transactions-repository-1-1f8dx) JGRP000029: transactions-repository-1-1f8dx: failed sending message to transactions-repository-1-4z05w (71 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=6259, TP: [cluster_name=cluster]
> [transactions-repository-1-04x09] 18:09:12,997 ERROR [org.jgroups.protocols.TCP] (jgroups-22,transactions-repository-1-04x09) JGRP000029: transactions-repository-1-04x09: failed sending message to transactions-repository-1-4z05w (71 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=5262, TP: [cluster_name=cluster]
> [transactions-repository-1-1f8dx] 18:09:13,113 ERROR [org.jgroups.protocols.TCP] (jgroups-16,transactions-repository-1-1f8dx) JGRP000029: transactions-repository-1-1f8dx: failed sending message to transactions-repository-1-4z05w (71 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=6259, TP: [cluster_name=cluster]
> {noformat}
> Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/530241bb695f1f490bcb25eabaf9d676
> Steps to reproduce:
> * Start local OpenShift Cluster
> * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer
> * invoke `cd transaction-creator && mvn fabric8:run`
> * Do the rolling update: `oc deploy transactions-repository --latest -n myproject`
> * Observe logs `kubetail -l environment=infrastructure`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 2 months
[JBoss JIRA] (ISPN-7489) org.jgroups.protocols.TCP emits errors when node leaves the cluster
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7489?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-7489:
--------------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4911
> org.jgroups.protocols.TCP emits errors when node leaves the cluster
> -------------------------------------------------------------------
>
> Key: ISPN-7489
> URL: https://issues.jboss.org/browse/ISPN-7489
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud Integrations, Core
> Affects Versions: 9.0.0.CR1
> Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}}
> * Custom Infinispan Server build (based on [these instructions|https://github.com/slaskawi/infinispan-1/tree/custom_image]). SHA1 {{2b0731b21649a88a75ed71d21b9cc06ba365e947}}
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Priority: Critical
>
> When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems.
> One of the is this:
> {noformat}
> [transactions-repository-1-04x09] 18:09:12,193 ERROR [org.jgroups.protocols.TCP] (jgroups-30,transactions-repository-1-04x09) JGRP000029: transactions-repository-1-04x09: failed sending message to transactions-repository-1-4z05w (71 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=5262, TP: [cluster_name=cluster]
> [transactions-repository-1-1f8dx] 18:09:12,310 ERROR [org.jgroups.protocols.TCP] (jgroups-16,transactions-repository-1-1f8dx) JGRP000029: transactions-repository-1-1f8dx: failed sending message to transactions-repository-1-4z05w (71 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=6259, TP: [cluster_name=cluster]
> [transactions-repository-1-04x09] 18:09:12,997 ERROR [org.jgroups.protocols.TCP] (jgroups-22,transactions-repository-1-04x09) JGRP000029: transactions-repository-1-04x09: failed sending message to transactions-repository-1-4z05w (71 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=5262, TP: [cluster_name=cluster]
> [transactions-repository-1-1f8dx] 18:09:13,113 ERROR [org.jgroups.protocols.TCP] (jgroups-16,transactions-repository-1-1f8dx) JGRP000029: transactions-repository-1-1f8dx: failed sending message to transactions-repository-1-4z05w (71 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=6259, TP: [cluster_name=cluster]
> {noformat}
> Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/530241bb695f1f490bcb25eabaf9d676
> Steps to reproduce:
> * Start local OpenShift Cluster
> * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer
> * invoke `cd transaction-creator && mvn fabric8:run`
> * Do the rolling update: `oc deploy transactions-repository --latest -n myproject`
> * Observe logs `kubetail -l environment=infrastructure`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 2 months
[JBoss JIRA] (ISPN-5728) Implement all the default methods in Java 8's Map interface
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5728?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-5728:
-------------------------------------
Assignee: William Burns
> Implement all the default methods in Java 8's Map interface
> -----------------------------------------------------------
>
> Key: ISPN-5728
> URL: https://issues.jboss.org/browse/ISPN-5728
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Affects Versions: 8.0.0.Final
> Reporter: Dan Berindei
> Assignee: William Burns
>
> Java 8 added many new methods to the {{Map}} interface. Some of them were already present in {{ConcurrentMap}}, but others are new: {{getOrDefault()}}, {{forEach()}}, {{replaceAll()}}, {{computeIfAbsent()}}, {{computeIfPresent()}}, {{compute()}}, {{merge()}}.
> We should try to write Infinispan-specific implementations wherever that would improve correctness and/or performance.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 2 months
[JBoss JIRA] (ISPN-5728) Implement all the default methods in Java 8's Map interface
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5728?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5728:
----------------------------------
Fix Version/s: (was: 9.0.0.Final)
> Implement all the default methods in Java 8's Map interface
> -----------------------------------------------------------
>
> Key: ISPN-5728
> URL: https://issues.jboss.org/browse/ISPN-5728
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Affects Versions: 8.0.0.Final
> Reporter: Dan Berindei
> Assignee: William Burns
>
> Java 8 added many new methods to the {{Map}} interface. Some of them were already present in {{ConcurrentMap}}, but others are new: {{getOrDefault()}}, {{forEach()}}, {{replaceAll()}}, {{computeIfAbsent()}}, {{computeIfPresent()}}, {{compute()}}, {{merge()}}.
> We should try to write Infinispan-specific implementations wherever that would improve correctness and/or performance.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 2 months