[JBoss JIRA] (ISPN-10136) Spurious dependency cycle detected error
by Dan Berindei (Jira)
Dan Berindei created ISPN-10136:
-----------------------------------
Summary: Spurious dependency cycle detected error
Key: ISPN-10136
URL: https://issues.jboss.org/browse/ISPN-10136
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 10.0.0.Beta3, 9.4.6.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 10.0.0.Beta4
{{BasicComponentRegistryImpl}} keeps track of which threads are currently wiring or starting components in order to make the error messages more user-friendly. In rare cases, however, the tracking information is not updated after a failure, and a spurious dependency cycle is logged:
{noformat}
14:56:09,073 WARN [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (jgroups-30,vlhebwmpr04-infinispan1) ISPN000071: Caught exception when handling command SingleRpcCommand{cacheName='offer-templates', command=PutKeyValueCommand{key=WrappedByteArray{bytes=[B0x0101290B033E0931..[16], hashCode=-798243737}, value=WrappedByteArray{bytes=[B0x01012A2962030409..[10856], hashCode=0}, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:vlkrbwmpr01-infinispan1:1968005, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=162205431146, maxIdle=-1, version=NumericVersion{version=159314845408592057}}, successful=true, topologyId=2329}}: org.infinispan.commons.CacheConfigurationException: Dependency cycle detected, please use ComponentRef<T> to break the cycle in path org.infinispan.interceptors.AsyncInterceptorChain (a org.infinispan.interceptors.impl.AsyncInterceptorChainImpl)
<< org.infinispan.expiration.impl.InternalExpirationManager (a org.infinispan.expiration.impl.ClusterExpirationManager)
<< org.infinispan.container.impl.InternalDataContainer (a org.infinispan.container.impl.BoundedSegmentedDataContainer)
<< org.infinispan.commands.CommandsFactory (a org.infinispan.commands.CommandsFactoryImpl)
<< org.infinispan.distribution.L1Manager (a org.infinispan.distribution.impl.L1ManagerImpl)
<< org.infinispan.distribution.RemoteValueRetrievedListener (a org.infinispan.factories.impl.ComponentAlias)
<< org.infinispan.interceptors.distribution.NonTxDistributionInterceptor (a org.infinispan.interceptors.distribution.NonTxDistributionInterceptor)
<< org.infinispan.interceptors.AsyncInterceptorChain (a org.infinispan.interceptors.impl.AsyncInterceptorChainImpl)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.awaitWrapperState(BasicComponentRegistryImpl.java:646)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:498)
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:714)
at org.infinispan.commands.CommandsFactoryImpl.initializeReplicableCommand(CommandsFactoryImpl.java:397)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.initializeCacheRpcCommand(GlobalInboundInvocationHandler.java:127)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:119)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
{noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years
[JBoss JIRA] (ISPN-10133) ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10133?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10133:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 10.0.0.Beta4
9.4.13.Final
Resolution: Done
> ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
> --------------------------------------------------------------------------------------
>
> Key: ISPN-10133
> URL: https://issues.jboss.org/browse/ISPN-10133
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.12.Final, 10.0.0.Beta3
> Environment: openjdk11-windows2012
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 10.0.0.Beta4, 9.4.13.Final
>
>
> We can reproduce the issue running the job https://jdg-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/JDG-2755-jdg-func...
> I attached the full log because I am not sure that this is the root cause. Feel free to add a comment.
> {noformat}
> The ' characters around the executable and arguments are
> not part of the command.
> [java] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djboss.modules.system.pkgs=com.sun.crypto.provider
> [java] Exception in thread "main" org.jboss.modules.ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
> [java] at org.jboss.modules.ModuleLoadException.toError(ModuleLoadException.java:74)
> [java] at org.jboss.modules.Module.getPathsUnchecked(Module.java:1608)
> [java] at org.jboss.modules.Module.loadModuleClass(Module.java:726)
> [java] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:247)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> [java] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> [java] at java.base/java.lang.ClassLoader.defineClass1(Native Method)
> [java] at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
> [java] at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:423)
> [java] at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:519)
> [java] at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:339)
> [java] at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:126)
> [java] at org.jboss.modules.Module.loadModuleClass(Module.java:731)
> [java] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:247)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> [java] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> [java] at java.base/java.lang.Class.forName0(Native Method)
> [java] at java.base/java.lang.Class.forName(Class.java:398)
> [java] at org.jboss.modules.Module.run(Module.java:338)
> [java] at org.jboss.modules.Module.run(Module.java:320)
> [java] at org.jboss.modules.Main.main(Main.java:593)
> [ant] Exiting C:\home\jenkins\workspace\JDG-AAAA-jdg-func-ispn-testsuite-reproducer\0ab58ada\infinispan\server\integration\testsuite\build-testsuite.xml.
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years
[JBoss JIRA] (ISPN-9291) BasePartitionHandlingTest.Partition.installMergeView() doesn't compute the merge digest
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9291?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9291:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/6853
* Force STABLE gc before merge to reproduce the problem
* Compute the NAKACK2 digest for the merge view to fix it
> BasePartitionHandlingTest.Partition.installMergeView() doesn't compute the merge digest
> ---------------------------------------------------------------------------------------
>
> Key: ISPN-9291
> URL: https://issues.jboss.org/browse/ISPN-9291
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.3.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
> Labels: testsuite_stability
> Fix For: 10.0.0.Final
>
>
> The partition handling tests use {{BasePartitionHandlingTest.Partition.installMergeView(view1, view2)}} to install the merge view without waiting for {{MERGE3}} to run, making them much faster. Unfortunately, the implementation is incorrect: {{GMS.installView(view)}} only works for regular views, merge views need to be installed with {{GMS.installView(mergeView, digest)}}.
> The result is that the nodes that got isolated from the coordinator request the retransmission of all the {{NAKACK2}} messages (including view updates) since the cluster first started. The isolated nodes cannot install the merge view until they deliver all the older messages (even without knowing whether they're OOB or not). But if {{STABLE}} ran and cleared a range of messages already, the retransmission request cannot be satisfied, so the view updates will never be delivered.
> This is easily reproducible in {{CrashedNodeDuringConflictResolutionTest}} if we add a delay before updating the topology in {{StateConsumerImpl}}. The test installs the merge view manually, but then kills NodeC and expects the cluster to install the new view automatically. NodeD can't install the new view because it's waiting for earlier messages from NodeA:
> {noformat}
> 18:27:13,054 INFO (testng-test:[]) [TestSuiteProgress] Test starting: org.infinispan.conflict.impl.CrashedNodeDuringConflictResolutionTest.testPartitionMergePolicy[DIST_SYNC]
> 18:27:13,640 DEBUG (testng-test:[]) [GMS] test-NodeA-39513: installing view MergeView::[test-NodeA-39513|10] (4) [test-NodeA-39513, test-NodeB-9439, test-NodeC-43706, test-NodeD-59078], 2 subgroups: [test-NodeA-39513|8] (2) [test-NodeA-39513, test-NodeB-9439], [test-NodeC-43706|9] (2) [test-NodeC-43706, test-NodeD-59078]
> 18:27:13,674 DEBUG (testng-test:[]) [GMS] test-NodeD-59078: installing view MergeView::[test-NodeA-39513|10] (4) [test-NodeA-39513, test-NodeB-9439, test-NodeC-43706, test-NodeD-59078], 2 subgroups: [test-NodeA-39513|8] (2) [test-NodeA-39513, test-NodeB-9439], [test-NodeC-43706|9] (2) [test-NodeC-43706, test-NodeD-59078]
> 18:27:13,828 TRACE (jgroups-7,test-NodeD-59078:[]) [NAKACK2] test-NodeD-59078: sending XMIT_REQ ((1): {50}) to test-NodeA-39513
> 18:27:13,966 TRACE (Timer runner-1,test-NodeD-59078:[]) [NAKACK2] test-NodeD-59078: sending XMIT_REQ ((49): {1-49}) to test-NodeA-39513
> 18:27:14,067 TRACE (Timer runner-1,test-NodeD-59078:[]) [NAKACK2] test-NodeD-59078: sending XMIT_REQ ((45): {1-45}) to test-NodeA-39513
> 18:27:14,504 DEBUG (testng-test:[]) [DefaultCacheManager] Stopping cache manager ISPN on test-NodeC-43706
> 18:27:18,642 TRACE (VERIFY_SUSPECT.TimerThread-89,test-NodeA-39513:[]) [GMS] test-NodeA-39513: joiners=[], suspected=[test-NodeC-43706], leaving=[], new view: [test-NodeA-39513|11] (3) [test-NodeA-39513, test-NodeB-9439, test-NodeD-59078]
> 18:27:18,643 TRACE (VERIFY_SUSPECT.TimerThread-89,test-NodeA-39513:[]) [GMS] test-NodeA-39513: mcasting view [test-NodeA-39513|11] (3) [test-NodeA-39513, test-NodeB-9439, test-NodeD-59078]
> 18:27:18,646 DEBUG (VERIFY_SUSPECT.TimerThread-89,test-NodeA-39513:[]) [GMS] test-NodeA-39513: installing view [test-NodeA-39513|11] (3) [test-NodeA-39513, test-NodeB-9439, test-NodeD-59078]
> 18:27:18,652 TRACE (VERIFY_SUSPECT.TimerThread-89,test-NodeA-39513:[]) [TCP_NIO2] test-NodeA-39513: sending msg to null, src=test-NodeA-39513, headers are GMS: GmsHeader[VIEW], NAKACK2: [MSG, seqno=63], TP: [cluster_name=ISPN]
> 18:27:18,656 TRACE (jgroups-20,test-NodeA-39513:[]) [TCP_NIO2] test-NodeA-39513: received [dst: test-NodeA-39513, src: test-NodeB-9439 (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=100, TP: [cluster_name=ISPN]
> 18:27:20,554 TRACE (Timer runner-1,test-NodeD-59078:[]) [NAKACK2] test-NodeD-59078: sending XMIT_REQ ((45): {1-45}) to test-NodeA-39513
> 18:27:20,653 WARN (VERIFY_SUSPECT.TimerThread-89,test-NodeA-39513:[]) [GMS] test-NodeA-39513: failed to collect all ACKs (expected=2) for view [test-NodeA-39513|11] after 2000ms, missing 1 ACKs from (1) test-NodeD-59078
> 18:27:20,656 TRACE (Timer runner-1,test-NodeD-59078:[]) [NAKACK2] test-NodeD-59078: sending XMIT_REQ ((45): {1-45}) to test-NodeA-39513
> 18:27:20,756 TRACE (Timer runner-1,test-NodeD-59078:[]) [NAKACK2] test-NodeD-59078: sending XMIT_REQ ((45): {1-45}) to test-NodeA-39513
> ...
> 18:28:14,412 TRACE (Timer runner-1,test-NodeD-59078:[]) [NAKACK2] test-NodeD-59078: sending XMIT_REQ ((45): {1-45}) to test-NodeA-39513
> 18:28:14,513 TRACE (Timer runner-1,test-NodeD-59078:[]) [NAKACK2] test-NodeD-59078: sending XMIT_REQ ((45): {1-45}) to test-NodeA-39513
> 18:28:14,589 ERROR (testng-test:[]) [TestSuiteProgress] Test failed: org.infinispan.conflict.impl.CrashedNodeDuringConflictResolutionTest.testPartitionMergePolicy[DIST_SYNC]
> java.lang.RuntimeException: Cache ___defaultcache timed out waiting for rebalancing to complete on node test-NodeA-39513, current topology is CacheTopology{id=21, phase=CONFLICT_RESOLUTION, rebalanceId=7, currentCH=PartitionerConsistentHash:DefaultConsistentHash{ns=256, owners = (3)[test-NodeD-59078: 256+0, test-NodeA-39513: 0+256, test-NodeB-9439: 0+256]}, pendingCH=null, unionCH=null, actualMembers=[test-NodeD-59078, test-NodeA-39513, test-NodeB-9439], persistentUUIDs=[828108c4-4251-49fc-9481-ff6392bea9fb, 1d4b6f07-b71b-41a1-adfb-abbe68944a9f, 3a1ece05-c282-433e-9eb5-7b3e0f1932aa]}. rebalanceInProgress=true, currentChIsBalanced=true
> at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:392) ~[test-classes/:?]
> at org.infinispan.conflict.impl.CrashedNodeDuringConflictResolutionTest.performMerge(CrashedNodeDuringConflictResolutionTest.java:113) ~[test-classes/:?]
> at org.infinispan.conflict.impl.BaseMergePolicyTest.testPartitionMergePolicy(BaseMergePolicyTest.java:137) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years