[JBoss JIRA] (ISPN-9988) ScatteredStateConsumerImpl can leak the exclusive topology lock
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9988?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9988:
----------------------------------
Fix Version/s: 10.0.1.Final
(was: 10.0.0.Final)
> ScatteredStateConsumerImpl can leak the exclusive topology lock
> ---------------------------------------------------------------
>
> Key: ISPN-9988
> URL: https://issues.jboss.org/browse/ISPN-9988
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.7.Final, 10.0.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.1.Final
>
>
> When an exception happens in {{ScatteredStateConsumerImpl.beforeTopologyInstalled}}, the exclusive topology lock is not released in {{StateConsumerImpl.onTopologyUpdate}}:
> {noformat}
> 15:21:54,783 ERROR (transport-thread-FunctionalScatteredInMemoryTest-NodeA-p43135-t5:[Topology-scattered]) [LocalTopologyManagerImpl] ISPN000230: Failed to start rebalance for cache scattered
> java.lang.IllegalArgumentException: The task is already cancelled.
> at org.infinispan.statetransfer.InboundTransferTask.cancelSegments(InboundTransferTask.java:172) ~[classes/:?]
> at org.infinispan.statetransfer.StateConsumerImpl.cancelTransfers(StateConsumerImpl.java:959) ~[classes/:?]
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.beforeTopologyInstalled(ScatteredStateConsumerImpl.java:115) ~[classes/:?]
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:292) ~[classes/:?]
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:102) ~[classes/:?]
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:200) ~[classes/:?]
> {noformat}
> Because the exclusive topology lock is not released, threads that try to apply a new topology update block forever. This causes random failures with the ISPN-9863 thread leak checker:
> {noformat}
> 15:26:25,922 WARN (testng-RehashClusterPublisherManagerTest:[]) [ThreadLeakChecker] Possible leaked thread:
> "transport-thread-FunctionalScatteredInMemoryTest-NodeA-p43135-t3" daemon prio=5 tid=0x236fd nid=NA waiting
> java.lang.Thread.State: WAITING
> java.base(a)11/jdk.internal.misc.Unsafe.park(Native Method)
> java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
> java.base@11/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:959)
> app//org.infinispan.statetransfer.StateTransferLockImpl.acquireExclusiveTopologyLock(StateTransferLockImpl.java:42)
> app//org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:291)
> app//org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:102)
> app//org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:200)
> app//org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:57)
> app//org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:113)
> app//org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:353)
> app//org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$1(LocalTopologyManagerImpl.java:275)
> 15:26:25,923 ERROR (testng-RehashClusterPublisherManagerTest:[]) [TestSuiteProgress] Test configuration failed: org.infinispan.reactive.publisher.impl.RehashClusterPublisherManagerTest.testClassFinished
> java.lang.AssertionError: Leaked threads:
> {transport-thread-FunctionalScatteredInMemoryTest-NodeA-p43135-t3: possible sources [org.infinispan.functional.FunctionalScatteredInMemoryTest[bias=ON_WRITE], org.infinispan.statetransfer.ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false], org.infinispan.functional.FunctionalCachestoreTest[passivation=true], org.infinispan.functional.distribution.rehash.FunctionalNonTxBackupOwnerBecomingPrimaryOwnerTest, org.infinispan.functional.distribution.rehash.FunctionalNonTxJoinerBecomingBackupOwnerTest, org.infinispan.api.mvcc.PutForExternalReadTest[REPL_SYNC, tx=false], org.infinispan.functional.distribution.rehash.FunctionalTxTest, org.infinispan.functional.FunctionalEncodingTypeTest[tx=true]]}
> at org.infinispan.commons.test.ThreadLeakChecker.performCheck(ThreadLeakChecker.java:148) ~[infinispan-commons-test-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at org.infinispan.commons.test.ThreadLeakChecker.testFinished(ThreadLeakChecker.java:109) ~[infinispan-commons-test-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at org.infinispan.test.fwk.TestResourceTracker.testFinished(TestResourceTracker.java:112) ~[test-classes/:?]
> at org.infinispan.test.AbstractInfinispanTest.testClassFinished(AbstractInfinispanTest.java:142) ~[test-classes/:?]
> {noformat}
> The fix should address both the exclusive topology lock itself, by releasing it in a finally block, and the {{IllegalArgumentException}}, either by ignoring already cancelled transfers or by only cancelling transfers while holding {{transferMapsLock}}.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
[JBoss JIRA] (ISPN-9917) JCachingProvider should not use weak ClassLoader references
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9917?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9917:
----------------------------------
Fix Version/s: 10.0.1.Final
(was: 10.0.0.Final)
> JCachingProvider should not use weak ClassLoader references
> -----------------------------------------------------------
>
> Key: ISPN-9917
> URL: https://issues.jboss.org/browse/ISPN-9917
> Project: Infinispan
> Issue Type: Bug
> Components: JCache
> Affects Versions: 10.0.0.Alpha3, 9.4.6.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.1.Final
>
>
> The idea behind {{WeakClassLoader}} is that we want users to be able to use {{Caching.getCachingProvider().getCacheManager()}} and not worry about closing the cache manager: instead Infinispan will close the manager when there are no more references to the provided {{ClassLoader}}.
> In order to close the manager automatically, {{AbstractJCachingProvider}} maintains a {{WeakHashMap}} of classloaders to managers, and if the cache manager had a strong reference to the classloader, the {{WeakHashMap}} entry would never be collected. However, this approach causes problems when a test creates a temporary {{ClassLoader}} instance and calls {{Caching.getCachingProvider().getCacheManager(tempClassLoader)}}, because the JVM is free to collect {{tempClassLoader}} immediately after it's used to create the {{WeakClassLoader}}. This can cause failures in the jcache tck with the IBM JDK:
> {noformat}
> [OK: 268, KO: 3, SKIP: 0] Test failed: CachingProviderTest.closeCacheManagerByURIAndClassLoader
> org.infinispan.commons.CacheConfigurationException: org.infinispan.commons.CacheException: Unable to construct a GlobalComponentRegistry!
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:344)
> at org.infinispan.jcache.embedded.JCacheManager.<init>(JCacheManager.java:70)
> at org.infinispan.jcache.embedded.JCachingProvider.createCacheManager(JCachingProvider.java:46)
> at org.infinispan.jcache.AbstractJCachingProvider.getCacheManager(AbstractJCachingProvider.java:67)
> at org.jsr107.tck.spi.CachingProviderTest.closeCacheManagerByURIAndClassLoader(CachingProviderTest.java:175)
> Caused by: org.infinispan.commons.CacheException: Unable to construct a GlobalComponentRegistry!
> at org.infinispan.factories.GlobalComponentRegistry.<init>(GlobalComponentRegistry.java:164)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:329)
> ... 39 more
> Caused by: org.infinispan.commons.CacheConfigurationException: Unable to inject dependencies for component class org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl, path org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifier (a org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performInjection(BasicComponentRegistryImpl.java:286)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.wireWrapper(BasicComponentRegistryImpl.java:176)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.registerComponent(BasicComponentRegistryImpl.java:360)
> at org.infinispan.factories.GlobalComponentRegistry.<init>(GlobalComponentRegistry.java:120)
> ... 40 more
> Caused by: org.infinispan.commons.CacheException: ClassLoader reference was garbage collected
> at org.infinispan.jcache.embedded.WeakClassLoader.requireClassLoader(WeakClassLoader.java:49)
> at org.infinispan.jcache.embedded.WeakClassLoader.findClass(WeakClassLoader.java:30)
> at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:925)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:870)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:853)
> at java.lang.Class.forNameImpl(Native Method)
> at java.lang.Class.forName(Class.java:402)
> at org.infinispan.commons.util.Util.loadClassStrict(Util.java:170)
> at org.infinispan.commons.util.Util.loadClass(Util.java:122)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.tryAutoInstantiation(BasicComponentRegistryImpl.java:249)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.findFactory(BasicComponentRegistryImpl.java:202)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.getComponent0(BasicComponentRegistryImpl.java:94)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.getDependency(BasicComponentRegistryImpl.java:327)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.setInjectionField(BasicComponentRegistryImpl.java:315)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performInjection(BasicComponentRegistryImpl.java:276)
> ... 43 more
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
[JBoss JIRA] (ISPN-9935) CDI RemoteCacheProducer leaks thread
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9935?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9935:
----------------------------------
Fix Version/s: 10.0.1.Final
(was: 10.0.0.Final)
> CDI RemoteCacheProducer leaks thread
> ------------------------------------
>
> Key: ISPN-9935
> URL: https://issues.jboss.org/browse/ISPN-9935
> Project: Infinispan
> Issue Type: Bug
> Components: CDI
> Affects Versions: 10.0.0.Alpha3, 9.4.6.Final
> Reporter: Dan Berindei
> Priority: Major
> Fix For: 10.0.1.Final
>
>
> {{RemoteCacheProducer}} injects an (optional?) default {{RemoteCacheManager}}, which is created with a custom bean registered by {{InfinispanExtensionRemote.createDefaultRemoteCacheManagerBean()}}.
> In the CDI remote tests, however, the default {{RemoteCacheManager}} is not used, so it's not created. Instead, Weld creates a proxy {{RemoteCacheManager}}, and since {{RemoteCacheManager}} is a class, its default constructor is invoked, connecting to the server and starting the async thread pool. But since it's not a proper bean, only a proxy, Weld never invokes {{stop()}} and leaks the thread.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months