[JBoss JIRA] (ISPN-9538) Allow for segmentation to be configurable via Features
by William Burns (JIRA)
William Burns created ISPN-9538:
-----------------------------------
Summary: Allow for segmentation to be configurable via Features
Key: ISPN-9538
URL: https://issues.jboss.org/browse/ISPN-9538
Project: Infinispan
Issue Type: Enhancement
Components: Core
Reporter: William Burns
Assignee: William Burns
Fix For: 9.4.0.Final
We want to allow the segmented containers to be able to be disabled for users where this is not supported. We recently added ISPN-9535 which allows for a nice easy way to do this. We just need to plug this into the DataContainerFactory class and have it enabled by default.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9537) Entry replaced with same expiration can expire immediately
by Wolf-Dieter Fink (JIRA)
Wolf-Dieter Fink created ISPN-9537:
--------------------------------------
Summary: Entry replaced with same expiration can expire immediately
Key: ISPN-9537
URL: https://issues.jboss.org/browse/ISPN-9537
Project: Infinispan
Issue Type: Bug
Components: Expiration
Affects Versions: 9.3.3.Final
Reporter: Wolf-Dieter Fink
Assignee: William Burns
Fix For: 9.4.0.Final
Currently lifespan expiration removes an expired entry based on the key value and lifespan parameter matching. Unfortunately this still leaves it open to removing an entry that was just inserted if it was expired and it was replaced with the same value.
This is especially problematic with `RemoteCache.putIfAbsent` or other conditional operations as they perform a get before doing the actual operation.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9536) Distributed stream iteration timeout during join
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9536?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9536:
-------------------------------
Status: Open (was: New)
> Distributed stream iteration timeout during join
> ------------------------------------------------
>
> Key: ISPN-9536
> URL: https://issues.jboss.org/browse/ISPN-9536
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.0.CR3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.4.0.Final
>
>
> {{LocalStreamManagerImpl}} checks in several places that the cache status is {{RUNNING}}, and if not it suspects all local segments with this log message:
> {noformat}
> [LocalStreamManagerImpl] Cache status is no longer running, all segments are now suspect for Test-NodeD-257370
> {noformat}
> This check is incorrect, because a node may receive distributed streaming requests before it is running: the initial state transfer could finish before all the components in the cache finish starting.
> If the check fails because the cache is still starting, the originator of the distributed stream iteration will assume that the remote cache has a newer topology. Because there is no newer topology, the originator blocks for {{DistributedCacheTimeout.timeout()}} and then fails.
> {noformat}
> 10:48:06,137 DEBUG (transport-thread-Test-NodeE-p660-t3:[Topology-org.infinispan.CONFIG]) [LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.CONFIG: CacheTopology{id=9, phase=NO_REBALANCE, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (5)[Test-NodeC-45735: 55, Test-NodeA-61957: 55, Test-NodeE-27687: 52, Test-NodeB-13641: 45, Test-NodeD-25737: 49]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeC-45735, Test-NodeA-61957, Test-NodeE-27687, Test-NodeB-13641, Test-NodeD-25737], persistentUUIDs=[5fccba8a-802a-423a-8e3b-ec9c1892a5ae, 5d04de93-79ce-4eff-a827-2a4bbdf54080, 70b79733-dbbe-4622-a5f6-38d3ec7771ff, 87e8c232-21a0-4416-8c81-4829f4619414, 78146ad7-7594-4161-9a7e-c4da0cc79f42]}
> 10:48:06,156 TRACE (remote-thread-Test-NodeE-p651-t2:[]) [LocalStreamManagerImpl] Request Test-NodeD-257370 completed for segments {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} with {} suspected segments
> 10:48:06,157 TRACE (remote-thread-Test-NodeE-p651-t2:[]) [LocalStreamManagerImpl] Cache status is no longer running, all segments are now suspect for Test-NodeD-257370
> 10:48:06,158 TRACE (ForkThread-5,CounterConcurrentStartTest:[org.infinispan.CONFIG]) [DefaultCacheManager] Cache org.infinispan.CONFIG started
> 10:48:06,165 TRACE (remote-thread-Test-NodeD-p649-t2:[]) [ClusterStreamManagerImpl] Received response from Test-NodeE-27687 with a completed response true for id Test-NodeD-257370 with {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} suspected segments.
> 10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [DistributedCacheStream] Found {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} lost segments for identifier Test-NodeD-257370
> 10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [DistributedCacheStream] Waiting for topology 10 to continue stream operation with segments {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255}
> 10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [StateTransferLockImpl] Waiting for topology 10 to be installed, current topology is 9
> 10:48:36,168 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [BasicComponentRegistryImpl] Changed status of org.infinispan.counter.api.CounterManager to FAILED
> org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.start() on object of type EmbeddedCounterManager
> ...
> Caused by: java.util.concurrent.TimeoutException
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_171]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_171]
> at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:330) ~[classes/:?]
> at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:229) ~[classes/:?]
> at org.infinispan.stream.impl.DistributedCacheStream.anyMatch(DistributedCacheStream.java:406) ~[classes/:?]
> at org.infinispan.util.AbstractDelegatingCacheStream.anyMatch(AbstractDelegatingCacheStream.java:300) ~[classes/:?]
> at org.infinispan.CacheStream.anyMatch(CacheStream.java:462) ~[classes/:?]
> at org.infinispan.counter.impl.manager.CounterConfigurationManager.start(CounterConfigurationManager.java:90) ~[classes/:?]
> at org.infinispan.counter.impl.manager.EmbeddedCounterManager.start(EmbeddedCounterManager.java:89) ~[classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9536) Distributed stream iteration timeout during join
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9536?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9536:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/6270
> Distributed stream iteration timeout during join
> ------------------------------------------------
>
> Key: ISPN-9536
> URL: https://issues.jboss.org/browse/ISPN-9536
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.0.CR3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.4.0.Final
>
>
> {{LocalStreamManagerImpl}} checks in several places that the cache status is {{RUNNING}}, and if not it suspects all local segments with this log message:
> {noformat}
> [LocalStreamManagerImpl] Cache status is no longer running, all segments are now suspect for Test-NodeD-257370
> {noformat}
> This check is incorrect, because a node may receive distributed streaming requests before it is running: the initial state transfer could finish before all the components in the cache finish starting.
> If the check fails because the cache is still starting, the originator of the distributed stream iteration will assume that the remote cache has a newer topology. Because there is no newer topology, the originator blocks for {{DistributedCacheTimeout.timeout()}} and then fails.
> {noformat}
> 10:48:06,137 DEBUG (transport-thread-Test-NodeE-p660-t3:[Topology-org.infinispan.CONFIG]) [LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.CONFIG: CacheTopology{id=9, phase=NO_REBALANCE, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (5)[Test-NodeC-45735: 55, Test-NodeA-61957: 55, Test-NodeE-27687: 52, Test-NodeB-13641: 45, Test-NodeD-25737: 49]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeC-45735, Test-NodeA-61957, Test-NodeE-27687, Test-NodeB-13641, Test-NodeD-25737], persistentUUIDs=[5fccba8a-802a-423a-8e3b-ec9c1892a5ae, 5d04de93-79ce-4eff-a827-2a4bbdf54080, 70b79733-dbbe-4622-a5f6-38d3ec7771ff, 87e8c232-21a0-4416-8c81-4829f4619414, 78146ad7-7594-4161-9a7e-c4da0cc79f42]}
> 10:48:06,156 TRACE (remote-thread-Test-NodeE-p651-t2:[]) [LocalStreamManagerImpl] Request Test-NodeD-257370 completed for segments {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} with {} suspected segments
> 10:48:06,157 TRACE (remote-thread-Test-NodeE-p651-t2:[]) [LocalStreamManagerImpl] Cache status is no longer running, all segments are now suspect for Test-NodeD-257370
> 10:48:06,158 TRACE (ForkThread-5,CounterConcurrentStartTest:[org.infinispan.CONFIG]) [DefaultCacheManager] Cache org.infinispan.CONFIG started
> 10:48:06,165 TRACE (remote-thread-Test-NodeD-p649-t2:[]) [ClusterStreamManagerImpl] Received response from Test-NodeE-27687 with a completed response true for id Test-NodeD-257370 with {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} suspected segments.
> 10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [DistributedCacheStream] Found {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} lost segments for identifier Test-NodeD-257370
> 10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [DistributedCacheStream] Waiting for topology 10 to continue stream operation with segments {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255}
> 10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [StateTransferLockImpl] Waiting for topology 10 to be installed, current topology is 9
> 10:48:36,168 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [BasicComponentRegistryImpl] Changed status of org.infinispan.counter.api.CounterManager to FAILED
> org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.start() on object of type EmbeddedCounterManager
> ...
> Caused by: java.util.concurrent.TimeoutException
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_171]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_171]
> at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:330) ~[classes/:?]
> at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:229) ~[classes/:?]
> at org.infinispan.stream.impl.DistributedCacheStream.anyMatch(DistributedCacheStream.java:406) ~[classes/:?]
> at org.infinispan.util.AbstractDelegatingCacheStream.anyMatch(AbstractDelegatingCacheStream.java:300) ~[classes/:?]
> at org.infinispan.CacheStream.anyMatch(CacheStream.java:462) ~[classes/:?]
> at org.infinispan.counter.impl.manager.CounterConfigurationManager.start(CounterConfigurationManager.java:90) ~[classes/:?]
> at org.infinispan.counter.impl.manager.EmbeddedCounterManager.start(EmbeddedCounterManager.java:89) ~[classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9527) Initial state transfer timed out for cache org.infinispan.CONFIG
by Michal Stehlik (JIRA)
[ https://issues.jboss.org/browse/ISPN-9527?page=com.atlassian.jira.plugin.... ]
Michal Stehlik commented on ISPN-9527:
--------------------------------------
We find out steps to reproduce in 3 different environments (each with 15 nodes). Each environment uses different switch HP or Mikrotik.
Trick is connect nodes sequentially every 13-15 seconds then previous one (since our nodes are slow and data which we will pushed into cluster contains cca 10K records per node it's sufficient delay).
Node marked as 151 was 1st joining to cluster. Log levels where set to DEBUG for Infinispan and JGroups.
What is quite strange is MergeView which I saw in one of JGroups logs because no network interruption was done during test.
Configuration for Infinispan and JGroups are also part of logs (see: master.zip).
> Initial state transfer timed out for cache org.infinispan.CONFIG
> ----------------------------------------------------------------
>
> Key: ISPN-9527
> URL: https://issues.jboss.org/browse/ISPN-9527
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud
> Affects Versions: 9.3.0.Final
> Environment: 15 nodes connecting to cloud in same time. ARM based processors, 1GHz processor, 1GB RAM. IPv4
> Reporter: Michal Stehlik
> Labels: timeout
> Attachments: cluster_config_distr.xml, infinispan.log, jgroupsObjectModel.xml, master.zip
>
>
> During starting of caches, several nodes fails due following exception (IPv4):
> {code:java}
> Cloud access error! org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:271) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ...
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on 0125f36a-2860-4107-810a-d087224c9105-39521
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9527) Initial state transfer timed out for cache org.infinispan.CONFIG
by Michal Stehlik (JIRA)
[ https://issues.jboss.org/browse/ISPN-9527?page=com.atlassian.jira.plugin.... ]
Michal Stehlik updated ISPN-9527:
---------------------------------
Attachment: master.zip
> Initial state transfer timed out for cache org.infinispan.CONFIG
> ----------------------------------------------------------------
>
> Key: ISPN-9527
> URL: https://issues.jboss.org/browse/ISPN-9527
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud
> Affects Versions: 9.3.0.Final
> Environment: 15 nodes connecting to cloud in same time. ARM based processors, 1GHz processor, 1GB RAM. IPv4
> Reporter: Michal Stehlik
> Labels: timeout
> Attachments: cluster_config_distr.xml, infinispan.log, jgroupsObjectModel.xml, master.zip
>
>
> During starting of caches, several nodes fails due following exception (IPv4):
> {code:java}
> Cloud access error! org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:271) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ...
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on 0125f36a-2860-4107-810a-d087224c9105-39521
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9536) Distributed stream iteration timeout during join
by Dan Berindei (JIRA)
Dan Berindei created ISPN-9536:
----------------------------------
Summary: Distributed stream iteration timeout during join
Key: ISPN-9536
URL: https://issues.jboss.org/browse/ISPN-9536
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 9.4.0.CR3
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 9.4.0.Final
{{LocalStreamManagerImpl}} checks in several places that the cache status is {{RUNNING}}, and if not it suspects all local segments with this log message:
{noformat}
[LocalStreamManagerImpl] Cache status is no longer running, all segments are now suspect for Test-NodeD-257370
{noformat}
This check is incorrect, because a node may receive distributed streaming requests before it is running: the initial state transfer could finish before all the components in the cache finish starting.
If the check fails because the cache is still starting, the originator of the distributed stream iteration will assume that the remote cache has a newer topology. Because there is no newer topology, the originator blocks for {{DistributedCacheTimeout.timeout()}} and then fails.
{noformat}
10:48:06,137 DEBUG (transport-thread-Test-NodeE-p660-t3:[Topology-org.infinispan.CONFIG]) [LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.CONFIG: CacheTopology{id=9, phase=NO_REBALANCE, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (5)[Test-NodeC-45735: 55, Test-NodeA-61957: 55, Test-NodeE-27687: 52, Test-NodeB-13641: 45, Test-NodeD-25737: 49]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeC-45735, Test-NodeA-61957, Test-NodeE-27687, Test-NodeB-13641, Test-NodeD-25737], persistentUUIDs=[5fccba8a-802a-423a-8e3b-ec9c1892a5ae, 5d04de93-79ce-4eff-a827-2a4bbdf54080, 70b79733-dbbe-4622-a5f6-38d3ec7771ff, 87e8c232-21a0-4416-8c81-4829f4619414, 78146ad7-7594-4161-9a7e-c4da0cc79f42]}
10:48:06,156 TRACE (remote-thread-Test-NodeE-p651-t2:[]) [LocalStreamManagerImpl] Request Test-NodeD-257370 completed for segments {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} with {} suspected segments
10:48:06,157 TRACE (remote-thread-Test-NodeE-p651-t2:[]) [LocalStreamManagerImpl] Cache status is no longer running, all segments are now suspect for Test-NodeD-257370
10:48:06,158 TRACE (ForkThread-5,CounterConcurrentStartTest:[org.infinispan.CONFIG]) [DefaultCacheManager] Cache org.infinispan.CONFIG started
10:48:06,165 TRACE (remote-thread-Test-NodeD-p649-t2:[]) [ClusterStreamManagerImpl] Received response from Test-NodeE-27687 with a completed response true for id Test-NodeD-257370 with {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} suspected segments.
10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [DistributedCacheStream] Found {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} lost segments for identifier Test-NodeD-257370
10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [DistributedCacheStream] Waiting for topology 10 to continue stream operation with segments {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255}
10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [StateTransferLockImpl] Waiting for topology 10 to be installed, current topology is 9
10:48:36,168 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [BasicComponentRegistryImpl] Changed status of org.infinispan.counter.api.CounterManager to FAILED
org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.start() on object of type EmbeddedCounterManager
...
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_171]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_171]
at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:330) ~[classes/:?]
at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:229) ~[classes/:?]
at org.infinispan.stream.impl.DistributedCacheStream.anyMatch(DistributedCacheStream.java:406) ~[classes/:?]
at org.infinispan.util.AbstractDelegatingCacheStream.anyMatch(AbstractDelegatingCacheStream.java:300) ~[classes/:?]
at org.infinispan.CacheStream.anyMatch(CacheStream.java:462) ~[classes/:?]
at org.infinispan.counter.impl.manager.CounterConfigurationManager.start(CounterConfigurationManager.java:90) ~[classes/:?]
at org.infinispan.counter.impl.manager.EmbeddedCounterManager.start(EmbeddedCounterManager.java:89) ~[classes/:?]
{noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years