[JBoss JIRA] (ISPN-11885) ChannelPool can get in inconsistent state with concurrent release and close of a channel
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11885?page=com.atlassian.jira.plugi... ]
Will Burns reassigned ISPN-11885:
---------------------------------
Assignee: Will Burns
> ChannelPool can get in inconsistent state with concurrent release and close of a channel
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-11885
> URL: https://issues.redhat.com/browse/ISPN-11885
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 10.1.8.Final
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 11.0.0.CR1
>
>
> While working on https://github.com/infinispan/infinispan/pull/8358 I found that the changes introduced allowed for a test failure to occur much more often. Upon investigation the issue is that the channel pool can close a channel and then release it which gets it in an inconsistent state.
> Looking closer it appears to be a bug in the ChannelPool class when a connection is closed before release completes.
> {code}
> 23:42:12,203 TRACE (DistributionRetryTest-Client-Async-1-2:[]) [ChannelPool] Closed channel [id: 0x27befd37, L:/127.0.0.1:37412 ! R:127.0.0.1/127.0.0.1:33943], created = 0, idle = false, active = 0
> 23:42:12,203 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Released channel [id: 0x27befd37, L:/127.0.0.1:37412 ! R:127.0.0.1/127.0.0.1:33943], active = -1
> {code}
> Here is the trace of it working where is is released before being closed.
> {code}
> 23:46:42,649 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Released channel [id: 0xeb9159ef, L:/127.0.0.1:39342 - R:127.0.0.1/127.0.0.1:41537], active = 0
> 23:46:42,792 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Activated record [id: 0x0b71d548, L:/127.0.0.1:51392 - R:127.0.0.1/127.0.0.1:37881], created = 1, active = 1
> 23:46:42,793 TRACE (DistributionRetryTest-Client-Async-1-3:[]) [ChannelPool] Closed channel [id: 0xeb9159ef, L:/127.0.0.1:39342 ! R:127.0.0.1/127.0.0.1:41537], created = 0, idle = true, active = 0
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 1 month
[JBoss JIRA] (ISPN-11885) ChannelPool can get in inconsistent state with concurrent release and close of a channel
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11885?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-11885:
------------------------------
Status: Open (was: New)
> ChannelPool can get in inconsistent state with concurrent release and close of a channel
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-11885
> URL: https://issues.redhat.com/browse/ISPN-11885
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 10.1.8.Final
> Reporter: Will Burns
> Priority: Major
> Fix For: 11.0.0.CR1
>
>
> While working on https://github.com/infinispan/infinispan/pull/8358 I found that the changes introduced allowed for a test failure to occur much more often. Upon investigation the issue is that the channel pool can close a channel and then release it which gets it in an inconsistent state.
> Looking closer it appears to be a bug in the ChannelPool class when a connection is closed before release completes.
> {code}
> 23:42:12,203 TRACE (DistributionRetryTest-Client-Async-1-2:[]) [ChannelPool] Closed channel [id: 0x27befd37, L:/127.0.0.1:37412 ! R:127.0.0.1/127.0.0.1:33943], created = 0, idle = false, active = 0
> 23:42:12,203 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Released channel [id: 0x27befd37, L:/127.0.0.1:37412 ! R:127.0.0.1/127.0.0.1:33943], active = -1
> {code}
> Here is the trace of it working where is is released before being closed.
> {code}
> 23:46:42,649 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Released channel [id: 0xeb9159ef, L:/127.0.0.1:39342 - R:127.0.0.1/127.0.0.1:41537], active = 0
> 23:46:42,792 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Activated record [id: 0x0b71d548, L:/127.0.0.1:51392 - R:127.0.0.1/127.0.0.1:37881], created = 1, active = 1
> 23:46:42,793 TRACE (DistributionRetryTest-Client-Async-1-3:[]) [ChannelPool] Closed channel [id: 0xeb9159ef, L:/127.0.0.1:39342 ! R:127.0.0.1/127.0.0.1:41537], created = 0, idle = true, active = 0
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 1 month
[JBoss JIRA] (ISPN-11885) ChannelPool can get in inconsistent state with concurrent release and close of a channel
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11885?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-11885:
------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/8358
> ChannelPool can get in inconsistent state with concurrent release and close of a channel
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-11885
> URL: https://issues.redhat.com/browse/ISPN-11885
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 10.1.8.Final
> Reporter: Will Burns
> Priority: Major
> Fix For: 11.0.0.CR1
>
>
> While working on https://github.com/infinispan/infinispan/pull/8358 I found that the changes introduced allowed for a test failure to occur much more often. Upon investigation the issue is that the channel pool can close a channel and then release it which gets it in an inconsistent state.
> Looking closer it appears to be a bug in the ChannelPool class when a connection is closed before release completes.
> {code}
> 23:42:12,203 TRACE (DistributionRetryTest-Client-Async-1-2:[]) [ChannelPool] Closed channel [id: 0x27befd37, L:/127.0.0.1:37412 ! R:127.0.0.1/127.0.0.1:33943], created = 0, idle = false, active = 0
> 23:42:12,203 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Released channel [id: 0x27befd37, L:/127.0.0.1:37412 ! R:127.0.0.1/127.0.0.1:33943], active = -1
> {code}
> Here is the trace of it working where is is released before being closed.
> {code}
> 23:46:42,649 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Released channel [id: 0xeb9159ef, L:/127.0.0.1:39342 - R:127.0.0.1/127.0.0.1:41537], active = 0
> 23:46:42,792 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Activated record [id: 0x0b71d548, L:/127.0.0.1:51392 - R:127.0.0.1/127.0.0.1:37881], created = 1, active = 1
> 23:46:42,793 TRACE (DistributionRetryTest-Client-Async-1-3:[]) [ChannelPool] Closed channel [id: 0xeb9159ef, L:/127.0.0.1:39342 ! R:127.0.0.1/127.0.0.1:41537], created = 0, idle = true, active = 0
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 1 month
[JBoss JIRA] (ISPN-11885) ChannelPool can get in inconsistent state with concurrent release and close of a channel
by Will Burns (Jira)
Will Burns created ISPN-11885:
---------------------------------
Summary: ChannelPool can get in inconsistent state with concurrent release and close of a channel
Key: ISPN-11885
URL: https://issues.redhat.com/browse/ISPN-11885
Project: Infinispan
Issue Type: Bug
Components: Hot Rod
Affects Versions: 10.1.8.Final
Reporter: Will Burns
Fix For: 11.0.0.CR1
While working on https://github.com/infinispan/infinispan/pull/8358 I found that the changes introduced allowed for a test failure to occur much more often. Upon investigation the issue is that the channel pool can close a channel and then release it which gets it in an inconsistent state.
Looking closer it appears to be a bug in the ChannelPool class when a connection is closed before release completes.
{code}
23:42:12,203 TRACE (DistributionRetryTest-Client-Async-1-2:[]) [ChannelPool] Closed channel [id: 0x27befd37, L:/127.0.0.1:37412 ! R:127.0.0.1/127.0.0.1:33943], created = 0, idle = false, active = 0
23:42:12,203 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Released channel [id: 0x27befd37, L:/127.0.0.1:37412 ! R:127.0.0.1/127.0.0.1:33943], active = -1
{code}
Here is the trace of it working where is is released before being closed.
{code}
23:46:42,649 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Released channel [id: 0xeb9159ef, L:/127.0.0.1:39342 - R:127.0.0.1/127.0.0.1:41537], active = 0
23:46:42,792 TRACE (testng-DistributionRetryTest:[]) [ChannelPool] Activated record [id: 0x0b71d548, L:/127.0.0.1:51392 - R:127.0.0.1/127.0.0.1:37881], created = 1, active = 1
23:46:42,793 TRACE (DistributionRetryTest-Client-Async-1-3:[]) [ChannelPool] Closed channel [id: 0xeb9159ef, L:/127.0.0.1:39342 ! R:127.0.0.1/127.0.0.1:41537], created = 0, idle = true, active = 0
{code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 1 month
[JBoss JIRA] (ISPN-11884) Exceptions when getting various attributes from a registered MBean
by David Maplesden (Jira)
David Maplesden created ISPN-11884:
--------------------------------------
Summary: Exceptions when getting various attributes from a registered MBean
Key: ISPN-11884
URL: https://issues.redhat.com/browse/ISPN-11884
Project: Infinispan
Issue Type: Bug
Components: JMX, reporting and management
Affects Versions: 10.1.8.Final
Reporter: David Maplesden
When trying to inspect some of the monitoring and reporting statistics exposed by infinispan via JMX we are getting an error retrieving some of the attributes.
The only objects we are getting errors from are of type {{org.infinispan.interceptors.impl.CacheMgmtInterceptor}}
The attributes:
* hitTimes
* missTimes
* storeTimes
* removeTimes
cause {{NullPointerException}} and the attribute
* statisticsEnabled
causes {{java.lang.IllegalArgumentException: wrong number of arguments}}
Debugging the issue it looks to me like the problem is with the way {{ResourceDMBean}} is discovering the getters for these attributes. For the first 4 attributes there are setters but no getter methods, so no method is found and we get the NPE. For the last attribute the code in {{ReflectionUtil.findGetterForFieldUsingReflection()}} just looks for a method with the correct name without checking the number of arguments, so it is finding the {{org.infinispan.interceptors.impl.CacheMgmtInterceptor.getStatisticsEnabled(FlagAffectedCommand)}} method instead of the {{org.infinispan.interceptors.impl.JmxStatsCommandInterceptor.getStatisticsEnabled()}} from the superclass.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 1 month
[JBoss JIRA] (ISPN-11882) LimitedExecutor.tryExecute blocking call causing random test failures
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11882?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-11882:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> LimitedExecutor.tryExecute blocking call causing random test failures
> ---------------------------------------------------------------------
>
> Key: ISPN-11882
> URL: https://issues.redhat.com/browse/ISPN-11882
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 11.0.0.Dev05
> Reporter: Dan Berindei
> Assignee: Will Burns
> Priority: Major
> Fix For: 11.0.0.CR1
>
>
> Causing random failures in {{ClusterTopologyManagerTest}}
> {noformat}
> java.lang.AssertionError: Blocking call! jdk.internal.misc.Unsafe#park on thread Thread[non-blocking-thread-ClusterTopologyManagerTest-NodeD-p15671-t5,5,ISPN-non-blocking-thread-group]
> at org.infinispan.util.CoreTestBlockHoundIntegration.lambda$applyTo$0(CoreTestBlockHoundIntegration.java:44)
> at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:383)
> at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89)
> at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
> at java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
> at org.infinispan.executors.LimitedExecutor.tryExecute(LimitedExecutor.java:142)
> at org.infinispan.executors.LimitedExecutor.access$300(LimitedExecutor.java:38)
> at org.infinispan.executors.LimitedExecutor$Runner.accept(LimitedExecutor.java:243)
> at org.infinispan.executors.LimitedExecutor$Runner.accept(LimitedExecutor.java:234)
> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:143)
> at org.infinispan.executors.LimitedExecutor.lambda$executeAsync$1(LimitedExecutor.java:136)
> at org.infinispan.executors.LimitedExecutor.actualRun(LimitedExecutor.java:192)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:176)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:38)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:237)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 1 month