[JBoss JIRA] (ISPN-9979) AbstractComponentRegistry.stop() can hang if called concurrently
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9979?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9979:
-------------------------------
Status: Open (was: New)
> AbstractComponentRegistry.stop() can hang if called concurrently
> ----------------------------------------------------------------
>
> Key: ISPN-9979
> URL: https://issues.jboss.org/browse/ISPN-9979
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.3.Final
> Reporter: Philippe Julien
> Assignee: Dan Berindei
> Priority: Major
>
> I believe that there is a bug in org.infinispan.factories.AbstractComponentRegistry.stop()
> Our Wildfly 15 nodes often hang on shutdown on the following:
> {noformat}
> "MSC service thread 1-2" #37 prio=5 os_prio=0 tid=0x0000000003807000 nid=0xf32d in Object.wait() [0x00007f0012c6f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005cc795fa8> (a org.infinispan.factories.ComponentRegistry)
> at java.lang.Object.wait(Object.java:502)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:359)
> - locked <0x00000005cc795fa8> (a org.infinispan.factories.ComponentRegistry)
> at org.infinispan.cache.impl.CacheImpl.performImmediateShutdown(CacheImpl.java:1159)
> at org.infinispan.cache.impl.CacheImpl.stop(CacheImpl.java:1124)
> at org.infinispan.cache.impl.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:520)
> at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:734)
> at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:786)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:762)
> at org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator.accept(CacheContainerServiceConfigurator.java:114)
> at org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator.accept(CacheContainerServiceConfigurator.java:70)
> at org.wildfly.clustering.service.FunctionalService.stop(FunctionalService.java:77)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1794)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.execute(ServiceControllerImpl.java:1763)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1558)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1364)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> There are no other threads that are working with 0x00000005cc795fa8 in the thread dump.
> I checked in a heap dump and the 0x00000005cc795fa8 org.infinispan.factories.ComponentRegistry object state is TERMINATED.
> I think that the finally block of AbstractComponentRegistry.stop() is missing a notifyAll().
> Shouldn't this:
> {code}
> ...
> } finally {
> synchronized (this) {
> state = ComponentStatus.TERMINATED;
> }
> }
> {code}
> Be:
> {code}
> ...
> } finally {
> synchronized (this) {
> state = ComponentStatus.TERMINATED;
> notifyAll();
> }
> }
> {code}
> This way, if two thread try to stop the same cache, the one that wins will notify the one that is waiting letting its stop complete.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 12 months
[JBoss JIRA] (ISPN-10136) Spurious dependency cycle detected error
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10136?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10136:
--------------------------------
Status: Open (was: New)
> Spurious dependency cycle detected error
> ----------------------------------------
>
> Key: ISPN-10136
> URL: https://issues.jboss.org/browse/ISPN-10136
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.6.Final, 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> {{BasicComponentRegistryImpl}} keeps track of which threads are currently wiring or starting components in order to make the error messages more user-friendly. In rare cases, however, the tracking information is not updated after a failure, and a spurious dependency cycle is logged:
> {noformat}
> 14:56:09,073 WARN [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (jgroups-30,vlhebwmpr04-infinispan1) ISPN000071: Caught exception when handling command SingleRpcCommand{cacheName='offer-templates', command=PutKeyValueCommand{key=WrappedByteArray{bytes=[B0x0101290B033E0931..[16], hashCode=-798243737}, value=WrappedByteArray{bytes=[B0x01012A2962030409..[10856], hashCode=0}, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:vlkrbwmpr01-infinispan1:1968005, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=162205431146, maxIdle=-1, version=NumericVersion{version=159314845408592057}}, successful=true, topologyId=2329}}: org.infinispan.commons.CacheConfigurationException: Dependency cycle detected, please use ComponentRef<T> to break the cycle in path org.infinispan.interceptors.AsyncInterceptorChain (a org.infinispan.interceptors.impl.AsyncInterceptorChainImpl)
> << org.infinispan.expiration.impl.InternalExpirationManager (a org.infinispan.expiration.impl.ClusterExpirationManager)
> << org.infinispan.container.impl.InternalDataContainer (a org.infinispan.container.impl.BoundedSegmentedDataContainer)
> << org.infinispan.commands.CommandsFactory (a org.infinispan.commands.CommandsFactoryImpl)
> << org.infinispan.distribution.L1Manager (a org.infinispan.distribution.impl.L1ManagerImpl)
> << org.infinispan.distribution.RemoteValueRetrievedListener (a org.infinispan.factories.impl.ComponentAlias)
> << org.infinispan.interceptors.distribution.NonTxDistributionInterceptor (a org.infinispan.interceptors.distribution.NonTxDistributionInterceptor)
> << org.infinispan.interceptors.AsyncInterceptorChain (a org.infinispan.interceptors.impl.AsyncInterceptorChainImpl)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.awaitWrapperState(BasicComponentRegistryImpl.java:646)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:498)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:714)
> at org.infinispan.commands.CommandsFactoryImpl.initializeReplicableCommand(CommandsFactoryImpl.java:397)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.initializeCacheRpcCommand(GlobalInboundInvocationHandler.java:127)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:119)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 12 months
[JBoss JIRA] (ISPN-10136) Spurious dependency cycle detected error
by Dan Berindei (Jira)
Dan Berindei created ISPN-10136:
-----------------------------------
Summary: Spurious dependency cycle detected error
Key: ISPN-10136
URL: https://issues.jboss.org/browse/ISPN-10136
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 10.0.0.Beta3, 9.4.6.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 10.0.0.Beta4
{{BasicComponentRegistryImpl}} keeps track of which threads are currently wiring or starting components in order to make the error messages more user-friendly. In rare cases, however, the tracking information is not updated after a failure, and a spurious dependency cycle is logged:
{noformat}
14:56:09,073 WARN [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (jgroups-30,vlhebwmpr04-infinispan1) ISPN000071: Caught exception when handling command SingleRpcCommand{cacheName='offer-templates', command=PutKeyValueCommand{key=WrappedByteArray{bytes=[B0x0101290B033E0931..[16], hashCode=-798243737}, value=WrappedByteArray{bytes=[B0x01012A2962030409..[10856], hashCode=0}, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:vlkrbwmpr01-infinispan1:1968005, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=162205431146, maxIdle=-1, version=NumericVersion{version=159314845408592057}}, successful=true, topologyId=2329}}: org.infinispan.commons.CacheConfigurationException: Dependency cycle detected, please use ComponentRef<T> to break the cycle in path org.infinispan.interceptors.AsyncInterceptorChain (a org.infinispan.interceptors.impl.AsyncInterceptorChainImpl)
<< org.infinispan.expiration.impl.InternalExpirationManager (a org.infinispan.expiration.impl.ClusterExpirationManager)
<< org.infinispan.container.impl.InternalDataContainer (a org.infinispan.container.impl.BoundedSegmentedDataContainer)
<< org.infinispan.commands.CommandsFactory (a org.infinispan.commands.CommandsFactoryImpl)
<< org.infinispan.distribution.L1Manager (a org.infinispan.distribution.impl.L1ManagerImpl)
<< org.infinispan.distribution.RemoteValueRetrievedListener (a org.infinispan.factories.impl.ComponentAlias)
<< org.infinispan.interceptors.distribution.NonTxDistributionInterceptor (a org.infinispan.interceptors.distribution.NonTxDistributionInterceptor)
<< org.infinispan.interceptors.AsyncInterceptorChain (a org.infinispan.interceptors.impl.AsyncInterceptorChainImpl)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.awaitWrapperState(BasicComponentRegistryImpl.java:646)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:498)
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:714)
at org.infinispan.commands.CommandsFactoryImpl.initializeReplicableCommand(CommandsFactoryImpl.java:397)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.initializeCacheRpcCommand(GlobalInboundInvocationHandler.java:127)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:119)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
{noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 12 months
[JBoss JIRA] (ISPN-10133) ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10133?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10133:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 10.0.0.Beta4
9.4.13.Final
Resolution: Done
> ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
> --------------------------------------------------------------------------------------
>
> Key: ISPN-10133
> URL: https://issues.jboss.org/browse/ISPN-10133
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.12.Final, 10.0.0.Beta3
> Environment: openjdk11-windows2012
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 10.0.0.Beta4, 9.4.13.Final
>
>
> We can reproduce the issue running the job https://jdg-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/JDG-2755-jdg-func...
> I attached the full log because I am not sure that this is the root cause. Feel free to add a comment.
> {noformat}
> The ' characters around the executable and arguments are
> not part of the command.
> [java] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djboss.modules.system.pkgs=com.sun.crypto.provider
> [java] Exception in thread "main" org.jboss.modules.ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
> [java] at org.jboss.modules.ModuleLoadException.toError(ModuleLoadException.java:74)
> [java] at org.jboss.modules.Module.getPathsUnchecked(Module.java:1608)
> [java] at org.jboss.modules.Module.loadModuleClass(Module.java:726)
> [java] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:247)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> [java] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> [java] at java.base/java.lang.ClassLoader.defineClass1(Native Method)
> [java] at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
> [java] at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:423)
> [java] at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:519)
> [java] at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:339)
> [java] at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:126)
> [java] at org.jboss.modules.Module.loadModuleClass(Module.java:731)
> [java] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:247)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> [java] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> [java] at java.base/java.lang.Class.forName0(Native Method)
> [java] at java.base/java.lang.Class.forName(Class.java:398)
> [java] at org.jboss.modules.Module.run(Module.java:338)
> [java] at org.jboss.modules.Module.run(Module.java:320)
> [java] at org.jboss.modules.Main.main(Main.java:593)
> [ant] Exiting C:\home\jenkins\workspace\JDG-AAAA-jdg-func-ispn-testsuite-reproducer\0ab58ada\infinispan\server\integration\testsuite\build-testsuite.xml.
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 12 months