6:42 a.m.
[
https://issues.jboss.org/browse/ISPN-9527?page=com.atlassian.jira.plugin....
]
Michal Stehlik commented on ISPN-9527:
--------------------------------------
We find out steps to reproduce in 3 different environments (each with 15 nodes). Each
environment uses different switch HP or Mikrotik.
Trick is connect nodes sequentially every 13-15 seconds then previous one (since our nodes
are slow and data which we will pushed into cluster contains cca 10K records per node
it's sufficient delay).
Node marked as 151 was 1st joining to cluster. Log levels where set to DEBUG for
Infinispan and JGroups.
What is quite strange is MergeView which I saw in one of JGroups logs because no network
interruption was done during test.
Configuration for Infinispan and JGroups are also part of logs (see: master.zip).
Initial state transfer timed out for cache org.infinispan.CONFIG
----------------------------------------------------------------
Key: ISPN-9527
URL: https://issues.jboss.org/browse/ISPN-9527
Project: Infinispan
Issue Type: Bug
Components: Cloud
Affects Versions: 9.3.0.Final
Environment: 15 nodes connecting to cloud in same time. ARM based processors,
1GHz processor, 1GB RAM. IPv4
Reporter: Michal Stehlik
Labels: timeout
Attachments: cluster_config_distr.xml, infinispan.log, jgroupsObjectModel.xml,
master.zip
During starting of caches, several nodes fails due following exception (IPv4):
{code:java}
Cloud access error! org.infinispan.commons.CacheException: Unable to invoke method public
void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object
of type GlobalConfigurationManagerImpl
org.infinispan.manager.EmbeddedCacheManagerStartupException:
org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of
type GlobalConfigurationManagerImpl
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:271)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
...
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of
type GlobalConfigurationManagerImpl
at
org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:44)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
... 33 more
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
at
org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:49)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
at
org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:44)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
... 33 more
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache org.infinispan.CONFIG on 0125f36a-2860-4107-810a-d087224c9105-39521
at
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
at
org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:49)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
at
org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:44)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267)
~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
... 33 more
{code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
2567
days inactive
2567
days old
infinispan-issues@lists.jboss.org
0 comments
1 participants
participants (1)
-
Michal Stehlik (JIRA)