[JBoss JIRA] (ISPN-10887) GlobalJmxStatisticsConfiguration.allowDuplicateDomains is not implemented atomically and can fail frequently
by Pedro Zapata Fernandez (Jira)
[ https://issues.redhat.com/browse/ISPN-10887?page=com.atlassian.jira.plugi... ]
Pedro Zapata Fernandez updated ISPN-10887:
------------------------------------------
Sprint: DataGrid Sprint #36, DataGrid Sprint #37, DataGrid Sprint #38 (was: DataGrid Sprint #36, DataGrid Sprint #37)
> GlobalJmxStatisticsConfiguration.allowDuplicateDomains is not implemented atomically and can fail frequently
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10887
> URL: https://issues.redhat.com/browse/ISPN-10887
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Affects Versions: 10.0.0.Final
> Reporter: Nistor Adrian
> Assignee: Nistor Adrian
> Priority: Major
> Fix For: 10.1.0.Beta1
>
>
> The feature is supposed to allow detection of an already occupied jmx domain and generate a new unique one by adding an increasing number suffix. The implementation polls the MBeanServer registry using queryNames until a free domain is found. The unique domain generation and the registration of the first MBean to occupy it is not atomic, so multiple CacheManagers starting up concurrently can mistakenly pick up the same domain.
> This can be fixed by making it atomic, by not using queryNames and instead directly registering the first bean, retrying if InstanceAlreadyExistsException. The first bean has to be picked identically by all cache managers. We'll pick first the cache manager itself and register the other components afterwards.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 4 months
[JBoss JIRA] (ISPN-10236) Hotrod client error releasing channel after server cache stop
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-10236?page=com.atlassian.jira.plugi... ]
Dan Berindei commented on ISPN-10236:
-------------------------------------
A new failure mode appeared recently:
{noformat}
org.infinispan.client.hotrod.exceptions.TransportException:: java.util.concurrent.RejectedExecutionException: Pool was terminated
at org.infinispan.client.hotrod.impl.Util.rewrap(Util.java:81)
at org.infinispan.client.hotrod.impl.Util.await(Util.java:54)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:344)
at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
at org.infinispan.client.hotrod.retry.ServerFailureRetryTest.testRetryCacheStopped(ServerFailureRetryTest.java:63)
at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.RejectedExecutionException: Pool was terminated
at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.close(ChannelPool.java:249)
at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.updateTopologyInfo(ChannelFactory.java:372)
at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.updateServers(ChannelFactory.java:318)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readNewTopologyAndHash(Codec20.java:416)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readNewTopologyIfPresent(Codec20.java:374)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:169)
at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:139)
at org.infinispan.client.hotrod.impl.transport.netty.HintedReplayingDecoder.callDecode(HintedReplayingDecoder.java:94)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
... 3 more
... Removed 18 stack frames
{noformat}
> Hotrod client error releasing channel after server cache stop
> -------------------------------------------------------------
>
> Key: ISPN-10236
> URL: https://issues.redhat.com/browse/ISPN-10236
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.1.0.Final
>
> Attachments: ISPN-10137_package_private_scope_20190524-1732_ServerFailureRetryTest-infinispan-client-hotrod.log.gz
>
>
> Random failure in {{ServerFailureRetryTest.testRetryCacheStopped}} caused by an assert statement in {{ChannelPool.release()}}.
> {noformat}
> 17:37:36,562 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.client.hotrod.retry.ServerFailureRetryTest.testRetryCacheStopped
> java.lang.AssertionError: Error releasing [id: 0x5d9755e6, L:/127.0.0.1:42472 ! R:127.0.0.1/127.0.0.1:44865]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.release(ChannelPool.java:170) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.releaseChannel(ChannelFactory.java:309) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.releaseChannel(HotRodOperation.java:105) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.invoke(RetryOnFailureOperation.java:80) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.activateChannel(ChannelPool.java:217) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.acquire(ChannelPool.java:86) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.fetchChannelAndInvoke(ChannelFactory.java:259) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.fetchChannelAndInvoke(ChannelFactory.java:297) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.fetchChannelAndInvoke(AbstractKeyOperation.java:41) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:61) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putAsync(RemoteCacheImpl.java:366) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:334) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79) ~[classes/:?]
> at org.infinispan.client.hotrod.retry.ServerFailureRetryTest.testRetryCacheStopped(ServerFailureRetryTest.java:63) ~[test-classes/:?]
> {noformat}
> I investigated a bit and I couldn't find an obvious mistake in the way {{ChannelPool.created}} is incremented and decremented, but I think it would help if access to it and {{ChannelPool.active}} were centralized in a smaller number of methods.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 4 months
[JBoss JIRA] (ISPN-10123) SecureServerFailureRetryTest random failures
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-10123?page=com.atlassian.jira.plugi... ]
Dan Berindei closed ISPN-10123.
-------------------------------
Fix Version/s: (was: 10.1.0.Final)
Resolution: Duplicate Issue
> SecureServerFailureRetryTest random failures
> --------------------------------------------
>
> Key: ISPN-10123
> URL: https://issues.redhat.com/browse/ISPN-10123
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 9.4.12.Final
> Reporter: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Attachments: SecureServerFailureRetryTest__home_dan_Downloads_infinispan-infinispan-client-hotrod.log2_20190409.zip, threaddump-org_infinispan_client_hotrod_retry_SecureServerFailureRetryTest_testRetryCacheStopped-2019-04-04-22303.log.xz
>
>
> {noformat}
> 06:32:49,516 DEBUG (testng-SecureServerFailureRetryTest:[]) [CacheImpl] Stopping cache ___defaultcache on SecureServerFailureRetryTest-NodeC-46394
> 06:32:49,524 WARN (SecureServerFailureRetryTest-Client-Async-111-1:[]) [ChannelFactory] ISPN004015: Failed adding new server 127.0.0.1:35206
> org.infinispan.client.hotrod.exceptions.RemoteIllegalLifecycleStateException: org.infinispan.IllegalLifecycleStateException: Cannot wire or start components while the registry is not running
> at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:344) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:177) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:138) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.HintedReplayingDecoder.callDecode(HintedReplayingDecoder.java:98) ~[classes/:?]
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.28.Final.jar:4.1.28.Final]
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.28.Final-linux-x86_64.jar:4.1.28.Final]
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:417) [netty-transport-native-epoll-4.1.28.Final-linux-x86_64.jar:4.1.28.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:317) [netty-transport-native-epoll-4.1.28.Final-linux-x86_64.jar:4.1.28.Final]
> [ERROR] Test org.infinispan.client.hotrod.retry.SecureServerFailureRetryTest.testRetryCacheStopped has been running for more than 300 seconds. Interrupting the test thread and dumping threads of the test suite process and its children.
> Dumping thread stacks of process 22303 to /home/infinispan/workspace/Infinispan_PR-6833/client/hotrod-client/threaddump-org_infinispan_client_hotrod_retry_SecureServerFailureRetryTest_testRetryCacheStopped-2019-04-04-22303.log
> Interrupted thread testng-SecureServerFailureRetryTest (21).
> [OK: 4016, KO: 1, SKIP: 0] Test failed: org.infinispan.client.hotrod.retry.SecureServerFailureRetryTest.testRetryCacheStopped
> org.infinispan.client.hotrod.exceptions.HotRodClientException:: java.lang.InterruptedException
> at org.infinispan.client.hotrod.impl.Util.await(Util.java:25)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:335)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at org.infinispan.client.hotrod.retry.ServerFailureRetryTest.testRetryCacheStopped(ServerFailureRetryTest.java:63)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 4 months
[JBoss JIRA] (ISPN-11065) Library replicated reads performance degradation
by Diego Lovison (Jira)
[ https://issues.redhat.com/browse/ISPN-11065?page=com.atlassian.jira.plugi... ]
Diego Lovison updated ISPN-11065:
---------------------------------
Security Sensitive Issue: (was: This issue is security relevant)
> Library replicated reads performance degradation
> ------------------------------------------------
>
> Key: ISPN-11065
> URL: https://issues.redhat.com/browse/ISPN-11065
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 10.1.0.Final
> Reporter: Diego Lovison
> Priority: Critical
>
> Library replicated Non-TX reads 10 threads
> ActualThroughput: -66.03 %
> ResponseTimeMean: +453.95 %
> Library replicated Non-TX reads 100 threads
> ActualThroughput: -67.31 %
> ResponseTimeMean: +582.92 %
> Library replicated TX reads 10 threads
> ActualThroughput: -12.63 %
> ResponseTimeMean: +15.41 %
> Library replicated TX reads 100 threads
> ActualThroughput: -12.14 %
> ResponseTimeMean: +14.25 %
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 4 months
[JBoss JIRA] (ISPN-11065) Library replicated reads performance degradation
by Diego Lovison (Jira)
[ https://issues.redhat.com/browse/ISPN-11065?page=com.atlassian.jira.plugi... ]
Diego Lovison updated ISPN-11065:
---------------------------------
Security Sensitive Issue: This issue is security relevant
> Library replicated reads performance degradation
> ------------------------------------------------
>
> Key: ISPN-11065
> URL: https://issues.redhat.com/browse/ISPN-11065
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 10.1.0.Final
> Reporter: Diego Lovison
> Priority: Critical
>
> Library replicated Non-TX reads 10 threads
> ActualThroughput: -66.03 %
> ResponseTimeMean: +453.95 %
> Library replicated Non-TX reads 100 threads
> ActualThroughput: -67.31 %
> ResponseTimeMean: +582.92 %
> Library replicated TX reads 10 threads
> ActualThroughput: -12.63 %
> ResponseTimeMean: +15.41 %
> Library replicated TX reads 100 threads
> ActualThroughput: -12.14 %
> ResponseTimeMean: +14.25 %
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 4 months
[JBoss JIRA] (ISPN-11065) Library replicated reads performance degradation
by Diego Lovison (Jira)
Diego Lovison created ISPN-11065:
------------------------------------
Summary: Library replicated reads performance degradation
Key: ISPN-11065
URL: https://issues.redhat.com/browse/ISPN-11065
Project: Infinispan
Issue Type: Bug
Affects Versions: 10.1.0.Final
Reporter: Diego Lovison
Library replicated Non-TX reads 10 threads
ActualThroughput: -66.03 %
ResponseTimeMean: +453.95 %
Library replicated Non-TX reads 100 threads
ActualThroughput: -67.31 %
ResponseTimeMean: +582.92 %
Library replicated TX reads 10 threads
ActualThroughput: -12.63 %
ResponseTimeMean: +15.41 %
Library replicated TX reads 100 threads
ActualThroughput: -12.14 %
ResponseTimeMean: +14.25 %
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 4 months
[JBoss JIRA] (ISPN-11065) Library replicated reads performance degradation
by Diego Lovison (Jira)
[ https://issues.redhat.com/browse/ISPN-11065?page=com.atlassian.jira.plugi... ]
Diego Lovison commented on ISPN-11065:
--------------------------------------
[~NadirX] [~afield] FYI
> Library replicated reads performance degradation
> ------------------------------------------------
>
> Key: ISPN-11065
> URL: https://issues.redhat.com/browse/ISPN-11065
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 10.1.0.Final
> Reporter: Diego Lovison
> Priority: Critical
>
> Library replicated Non-TX reads 10 threads
> ActualThroughput: -66.03 %
> ResponseTimeMean: +453.95 %
> Library replicated Non-TX reads 100 threads
> ActualThroughput: -67.31 %
> ResponseTimeMean: +582.92 %
> Library replicated TX reads 10 threads
> ActualThroughput: -12.63 %
> ResponseTimeMean: +15.41 %
> Library replicated TX reads 100 threads
> ActualThroughput: -12.14 %
> ResponseTimeMean: +14.25 %
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 4 months