[JBoss JIRA] (ISPN-9527) Initial state transfer timed out for cache org.infinispan.CONFIG
by Michal Stehlik (JIRA)
[ https://issues.jboss.org/browse/ISPN-9527?page=com.atlassian.jira.plugin.... ]
Michal Stehlik updated ISPN-9527:
---------------------------------
Attachment: master.zip
> Initial state transfer timed out for cache org.infinispan.CONFIG
> ----------------------------------------------------------------
>
> Key: ISPN-9527
> URL: https://issues.jboss.org/browse/ISPN-9527
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud
> Affects Versions: 9.3.0.Final
> Environment: 15 nodes connecting to cloud in same time. ARM based processors, 1GHz processor, 1GB RAM. IPv4
> Reporter: Michal Stehlik
> Labels: timeout
> Attachments: cluster_config_distr.xml, infinispan.log, jgroupsObjectModel.xml, master.zip
>
>
> During starting of caches, several nodes fails due following exception (IPv4):
> {code:java}
> Cloud access error! org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:271) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ...
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on 0125f36a-2860-4107-810a-d087224c9105-39521
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> ... 33 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9536) Distributed stream iteration timeout during join
by Dan Berindei (JIRA)
Dan Berindei created ISPN-9536:
----------------------------------
Summary: Distributed stream iteration timeout during join
Key: ISPN-9536
URL: https://issues.jboss.org/browse/ISPN-9536
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 9.4.0.CR3
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 9.4.0.Final
{{LocalStreamManagerImpl}} checks in several places that the cache status is {{RUNNING}}, and if not it suspects all local segments with this log message:
{noformat}
[LocalStreamManagerImpl] Cache status is no longer running, all segments are now suspect for Test-NodeD-257370
{noformat}
This check is incorrect, because a node may receive distributed streaming requests before it is running: the initial state transfer could finish before all the components in the cache finish starting.
If the check fails because the cache is still starting, the originator of the distributed stream iteration will assume that the remote cache has a newer topology. Because there is no newer topology, the originator blocks for {{DistributedCacheTimeout.timeout()}} and then fails.
{noformat}
10:48:06,137 DEBUG (transport-thread-Test-NodeE-p660-t3:[Topology-org.infinispan.CONFIG]) [LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.CONFIG: CacheTopology{id=9, phase=NO_REBALANCE, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (5)[Test-NodeC-45735: 55, Test-NodeA-61957: 55, Test-NodeE-27687: 52, Test-NodeB-13641: 45, Test-NodeD-25737: 49]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeC-45735, Test-NodeA-61957, Test-NodeE-27687, Test-NodeB-13641, Test-NodeD-25737], persistentUUIDs=[5fccba8a-802a-423a-8e3b-ec9c1892a5ae, 5d04de93-79ce-4eff-a827-2a4bbdf54080, 70b79733-dbbe-4622-a5f6-38d3ec7771ff, 87e8c232-21a0-4416-8c81-4829f4619414, 78146ad7-7594-4161-9a7e-c4da0cc79f42]}
10:48:06,156 TRACE (remote-thread-Test-NodeE-p651-t2:[]) [LocalStreamManagerImpl] Request Test-NodeD-257370 completed for segments {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} with {} suspected segments
10:48:06,157 TRACE (remote-thread-Test-NodeE-p651-t2:[]) [LocalStreamManagerImpl] Cache status is no longer running, all segments are now suspect for Test-NodeD-257370
10:48:06,158 TRACE (ForkThread-5,CounterConcurrentStartTest:[org.infinispan.CONFIG]) [DefaultCacheManager] Cache org.infinispan.CONFIG started
10:48:06,165 TRACE (remote-thread-Test-NodeD-p649-t2:[]) [ClusterStreamManagerImpl] Received response from Test-NodeE-27687 with a completed response true for id Test-NodeD-257370 with {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} suspected segments.
10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [DistributedCacheStream] Found {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255} lost segments for identifier Test-NodeD-257370
10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [DistributedCacheStream] Waiting for topology 10 to continue stream operation with segments {2 4 12 20 24 27 31 36 38 42 47 56 58 66 72 75 77 87-88 93 100-101 109 120 125 148 150 159 166 172 179-180 188 194-195 198 204-205 212 216 219-220 222 226 233 236-237 239 242-244 255}
10:48:06,165 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [StateTransferLockImpl] Waiting for topology 10 to be installed, current topology is 9
10:48:36,168 TRACE (ForkThread-9,CounterConcurrentStartTest:[]) [BasicComponentRegistryImpl] Changed status of org.infinispan.counter.api.CounterManager to FAILED
org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.start() on object of type EmbeddedCounterManager
...
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_171]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_171]
at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:330) ~[classes/:?]
at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:229) ~[classes/:?]
at org.infinispan.stream.impl.DistributedCacheStream.anyMatch(DistributedCacheStream.java:406) ~[classes/:?]
at org.infinispan.util.AbstractDelegatingCacheStream.anyMatch(AbstractDelegatingCacheStream.java:300) ~[classes/:?]
at org.infinispan.CacheStream.anyMatch(CacheStream.java:462) ~[classes/:?]
at org.infinispan.counter.impl.manager.CounterConfigurationManager.start(CounterConfigurationManager.java:90) ~[classes/:?]
at org.infinispan.counter.impl.manager.EmbeddedCounterManager.start(EmbeddedCounterManager.java:89) ~[classes/:?]
{noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9425) HotRod Transaction Cache - Wrong values when a transactional cache stops in the middle of PrepareTransactionOperation/CompleteTransactionOperation
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-9425?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-9425:
-----------------------------------
Scenario 1:
I need to do some digging about the failures in program 2. I'm not sure if there are any timeouts check for the operation execution time.
Program 1 is expected to fail. After 1 min, the server will rollback the transaction from program 1 and, when it tries to commit, the transaction no longer exists.
Scenario 2:
The scenario 2 is expected. When the program 2 completed, the transaction in program 1 isn't committed yet. So, the key's value is null.
Scenario 3:
The transaction in program 1 is expected to fail. I'm not sure where the assertion error comes from.
> HotRod Transaction Cache - Wrong values when a transactional cache stops in the middle of PrepareTransactionOperation/CompleteTransactionOperation
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9425
> URL: https://issues.jboss.org/browse/ISPN-9425
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Assignee: Pedro Ruivo
> Priority: Critical
>
> There are 3 scenarios and I believe that they are correlated. If it is not the case, a JIRA for each item should be open.
> All scenarios below are multiple clients and one server. The following steps should be executed does not matter what is the scenario.
> * Add one breakpoint at the first line of the method `acceptResponse` at the class PrepareTransactionOperation
> * Add one breakpoint at the first line of the method `executeOperation` at the class PrepareTransactionOperation
> *Scenario 1*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume. It will stop at `executeOperation`.
> * Wait 5 minutes or any time greater than 60 seconds ( I was not able to identify the right period of time but greater than 60 seconds should work), Resume and Resume the "Program 2". It will fail.
> * The issue in scenario 1 is: Why doesn't matter the amount of time since it is greater than 60 seconds the request will fail? I am expecting something like a timeout in the "Program 1" and not expire when a call in that key was called for the first time.
> * Kill all clients and servers
> *Scenario 2*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume, Resume, Resume. It will fail. Here you don't need to wait. Just press the Resume.
> * Program 1 -> Resume.
> * The issue in scenario 2 is: Why the test fails due: expected value 'null' different from 'random-value-generated'
> * Kill all clients and servers
> *Scenario 3*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume, Resume, Resume. It will fail. Here you don't need to wait. Just press the Resume.
> * Wait 60 seconds
> * Re-run Program 2 -> Resume, Resume, Resume, Resume. It will work.
> * Program 1 -> Resume, it will fail due to an assertion error
> * The issue in scenario 3 is: When resuming the "Program 1" I am expecting an error during the commit phase and not an assertion error.
> * Kill all clients and servers
> Program 1 means that you should run the client code. It will create a separated JVM.
> Program 2 means that you should run the client code. It will create a separated JVM.
> It will be nice if we find out an way to automate this kind of tests.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9425) HotRod Transaction Cache - Wrong values when a transactional cache stops in the middle of PrepareTransactionOperation/CompleteTransactionOperation
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-9425?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-9425:
-----------------------------------
Some explanations first...
Since it is impossible to know if the client has crashed, the server uses an idle timeout to rollback a transaction. If the client doesn't interact with that transaction in that period of time, the transaction is rolled back. By default, the timeout is 1min and it can be configured via https://github.com/infinispan/infinispan/blob/master/client/hotrod-client...
The explanation why you don't have {{TimeoutExceptions}} or similar is here: https://issues.jboss.org/browse/ISPN-9420?focusedCommentId=13637344&page=...
> HotRod Transaction Cache - Wrong values when a transactional cache stops in the middle of PrepareTransactionOperation/CompleteTransactionOperation
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9425
> URL: https://issues.jboss.org/browse/ISPN-9425
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Assignee: Pedro Ruivo
> Priority: Critical
>
> There are 3 scenarios and I believe that they are correlated. If it is not the case, a JIRA for each item should be open.
> All scenarios below are multiple clients and one server. The following steps should be executed does not matter what is the scenario.
> * Add one breakpoint at the first line of the method `acceptResponse` at the class PrepareTransactionOperation
> * Add one breakpoint at the first line of the method `executeOperation` at the class PrepareTransactionOperation
> *Scenario 1*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume. It will stop at `executeOperation`.
> * Wait 5 minutes or any time greater than 60 seconds ( I was not able to identify the right period of time but greater than 60 seconds should work), Resume and Resume the "Program 2". It will fail.
> * The issue in scenario 1 is: Why doesn't matter the amount of time since it is greater than 60 seconds the request will fail? I am expecting something like a timeout in the "Program 1" and not expire when a call in that key was called for the first time.
> * Kill all clients and servers
> *Scenario 2*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume, Resume, Resume. It will fail. Here you don't need to wait. Just press the Resume.
> * Program 1 -> Resume.
> * The issue in scenario 2 is: Why the test fails due: expected value 'null' different from 'random-value-generated'
> * Kill all clients and servers
> *Scenario 3*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume, Resume, Resume. It will fail. Here you don't need to wait. Just press the Resume.
> * Wait 60 seconds
> * Re-run Program 2 -> Resume, Resume, Resume, Resume. It will work.
> * Program 1 -> Resume, it will fail due to an assertion error
> * The issue in scenario 3 is: When resuming the "Program 1" I am expecting an error during the commit phase and not an assertion error.
> * Kill all clients and servers
> Program 1 means that you should run the client code. It will create a separated JVM.
> Program 2 means that you should run the client code. It will create a separated JVM.
> It will be nice if we find out an way to automate this kind of tests.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9420) HotRod Transaction Cache is throwing an exception with multiple threads
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-9420?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo reopened ISPN-9420:
-------------------------------
Assignee: Pedro Ruivo
> HotRod Transaction Cache is throwing an exception with multiple threads
> -----------------------------------------------------------------------
>
> Key: ISPN-9420
> URL: https://issues.jboss.org/browse/ISPN-9420
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Assignee: Pedro Ruivo
>
> I created the method `public void testMultiThreadsPessimisticLockMode(Method method)` in the class `TxFunctionalTest`
> {code:java}
> public void testMultiThreadsPessimisticLockMode(Method method) throws ExecutionException, InterruptedException {
> final K k1 = kvGenerator.generateKey(method, 1);
> final V v1 = kvGenerator.generateValue(method, 1);
> final V v2 = kvGenerator.generateValue(method, 2);
> RemoteCache<K, V> remoteCache = remoteCacheWithForceReturnValue();
> final TransactionManager tm = remoteCache.getTransactionManager();
> ExecutorService executorService = Executors.newFixedThreadPool(2);
> Future<?> f1 = executorService.submit(() -> {
> try {
> TestingUtil.withTx(tm, () -> {
> V returnValue = remoteCache.put(k1, v1);
> Thread.sleep(2000);
> return returnValue;
> });
> } catch (Exception e) {
> throw new IllegalStateException(e);
> }
> });
> Future<?> f2 = executorService.submit(() -> {
> try {
> TestingUtil.withTx(tm, () -> {
> Thread.sleep(500);
> V returnValue = remoteCache.put(k1, v2);
> return returnValue;
> });
> } catch (Exception e) {
> throw new IllegalStateException(e);
> }
> });
> Assert.assertNull(f1.get());
> Assert.assertNull(f2.get());
> }
> {code}
> The integration test is failing due:
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.testMultiThreadsPessimisticLockMode(TxFunctionalTest.java:473)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:196)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:208)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:635)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:816)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1124)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
> at org.testng.TestRunner.privateRun(TestRunner.java:774)
> at org.testng.TestRunner.run(TestRunner.java:624)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:359)
> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:354)
> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:312)
> at org.testng.SuiteRunner.run(SuiteRunner.java:261)
> at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
> at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
> at org.testng.TestNG.runSuitesSequentially(TestNG.java:1215)
> at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
> at org.testng.TestNG.run(TestNG.java:1048)
> at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
> at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
> Caused by: java.lang.IllegalStateException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.lambda$testMultiThreadsPessimisticLockMode$1(TxFunctionalTest.java:457)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
> at org.infinispan.test.TestingUtil.lambda$withTxCallable$11(TestingUtil.java:1529)
> at org.infinispan.test.TestingUtil.withTx(TestingUtil.java:1507)
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.lambda$testMultiThreadsPessimisticLockMode$1(TxFunctionalTest.java:451)
> ... 5 more
> Caused by: java.lang.Throwable: setRollbackOnly called from:
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.setRollbackOnly(TransactionImple.java:339)
> at org.infinispan.client.hotrod.impl.transaction.SyncModeTransactionTable$SynchronizationAdapter.markAsRollback(SyncModeTransactionTable.java:150)
> at org.infinispan.client.hotrod.impl.transaction.SyncModeTransactionTable$SynchronizationAdapter.beforeCompletion(SyncModeTransactionTable.java:125)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:371)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
> ... 9 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9420) HotRod Transaction Cache is throwing an exception with multiple threads
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-9420?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo closed ISPN-9420.
-----------------------------
Resolution: Explained
> HotRod Transaction Cache is throwing an exception with multiple threads
> -----------------------------------------------------------------------
>
> Key: ISPN-9420
> URL: https://issues.jboss.org/browse/ISPN-9420
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Assignee: Pedro Ruivo
>
> I created the method `public void testMultiThreadsPessimisticLockMode(Method method)` in the class `TxFunctionalTest`
> {code:java}
> public void testMultiThreadsPessimisticLockMode(Method method) throws ExecutionException, InterruptedException {
> final K k1 = kvGenerator.generateKey(method, 1);
> final V v1 = kvGenerator.generateValue(method, 1);
> final V v2 = kvGenerator.generateValue(method, 2);
> RemoteCache<K, V> remoteCache = remoteCacheWithForceReturnValue();
> final TransactionManager tm = remoteCache.getTransactionManager();
> ExecutorService executorService = Executors.newFixedThreadPool(2);
> Future<?> f1 = executorService.submit(() -> {
> try {
> TestingUtil.withTx(tm, () -> {
> V returnValue = remoteCache.put(k1, v1);
> Thread.sleep(2000);
> return returnValue;
> });
> } catch (Exception e) {
> throw new IllegalStateException(e);
> }
> });
> Future<?> f2 = executorService.submit(() -> {
> try {
> TestingUtil.withTx(tm, () -> {
> Thread.sleep(500);
> V returnValue = remoteCache.put(k1, v2);
> return returnValue;
> });
> } catch (Exception e) {
> throw new IllegalStateException(e);
> }
> });
> Assert.assertNull(f1.get());
> Assert.assertNull(f2.get());
> }
> {code}
> The integration test is failing due:
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.testMultiThreadsPessimisticLockMode(TxFunctionalTest.java:473)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:196)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:208)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:635)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:816)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1124)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
> at org.testng.TestRunner.privateRun(TestRunner.java:774)
> at org.testng.TestRunner.run(TestRunner.java:624)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:359)
> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:354)
> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:312)
> at org.testng.SuiteRunner.run(SuiteRunner.java:261)
> at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
> at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
> at org.testng.TestNG.runSuitesSequentially(TestNG.java:1215)
> at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
> at org.testng.TestNG.run(TestNG.java:1048)
> at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
> at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
> Caused by: java.lang.IllegalStateException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.lambda$testMultiThreadsPessimisticLockMode$1(TxFunctionalTest.java:457)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
> at org.infinispan.test.TestingUtil.lambda$withTxCallable$11(TestingUtil.java:1529)
> at org.infinispan.test.TestingUtil.withTx(TestingUtil.java:1507)
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.lambda$testMultiThreadsPessimisticLockMode$1(TxFunctionalTest.java:451)
> ... 5 more
> Caused by: java.lang.Throwable: setRollbackOnly called from:
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.setRollbackOnly(TransactionImple.java:339)
> at org.infinispan.client.hotrod.impl.transaction.SyncModeTransactionTable$SynchronizationAdapter.markAsRollback(SyncModeTransactionTable.java:150)
> at org.infinispan.client.hotrod.impl.transaction.SyncModeTransactionTable$SynchronizationAdapter.beforeCompletion(SyncModeTransactionTable.java:125)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:371)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
> ... 9 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9425) HotRod Transaction Cache - Wrong values when a transactional cache stops in the middle of PrepareTransactionOperation/CompleteTransactionOperation
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-9425?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo reassigned ISPN-9425:
---------------------------------
Assignee: Pedro Ruivo
> HotRod Transaction Cache - Wrong values when a transactional cache stops in the middle of PrepareTransactionOperation/CompleteTransactionOperation
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9425
> URL: https://issues.jboss.org/browse/ISPN-9425
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Assignee: Pedro Ruivo
> Priority: Critical
>
> There are 3 scenarios and I believe that they are correlated. If it is not the case, a JIRA for each item should be open.
> All scenarios below are multiple clients and one server. The following steps should be executed does not matter what is the scenario.
> * Add one breakpoint at the first line of the method `acceptResponse` at the class PrepareTransactionOperation
> * Add one breakpoint at the first line of the method `executeOperation` at the class PrepareTransactionOperation
> *Scenario 1*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume. It will stop at `executeOperation`.
> * Wait 5 minutes or any time greater than 60 seconds ( I was not able to identify the right period of time but greater than 60 seconds should work), Resume and Resume the "Program 2". It will fail.
> * The issue in scenario 1 is: Why doesn't matter the amount of time since it is greater than 60 seconds the request will fail? I am expecting something like a timeout in the "Program 1" and not expire when a call in that key was called for the first time.
> * Kill all clients and servers
> *Scenario 2*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume, Resume, Resume. It will fail. Here you don't need to wait. Just press the Resume.
> * Program 1 -> Resume.
> * The issue in scenario 2 is: Why the test fails due: expected value 'null' different from 'random-value-generated'
> * Kill all clients and servers
> *Scenario 3*
> * Execute the client code ( Program 1 )
> * Execute the client code ( Program 2 )
> * Program 1 -> Resume, Resume, Resume. It will stop at `acceptResponse`.
> * Program 2 -> Resume, Resume, Resume, Resume. It will fail. Here you don't need to wait. Just press the Resume.
> * Wait 60 seconds
> * Re-run Program 2 -> Resume, Resume, Resume, Resume. It will work.
> * Program 1 -> Resume, it will fail due to an assertion error
> * The issue in scenario 3 is: When resuming the "Program 1" I am expecting an error during the commit phase and not an assertion error.
> * Kill all clients and servers
> Program 1 means that you should run the client code. It will create a separated JVM.
> Program 2 means that you should run the client code. It will create a separated JVM.
> It will be nice if we find out an way to automate this kind of tests.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9420) HotRod Transaction Cache is throwing an exception with multiple threads
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-9420?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo closed ISPN-9420.
-----------------------------
Resolution: Explained
I'm going to close this for now. Re-Open it if you have more questions :)
> HotRod Transaction Cache is throwing an exception with multiple threads
> -----------------------------------------------------------------------
>
> Key: ISPN-9420
> URL: https://issues.jboss.org/browse/ISPN-9420
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
>
> I created the method `public void testMultiThreadsPessimisticLockMode(Method method)` in the class `TxFunctionalTest`
> {code:java}
> public void testMultiThreadsPessimisticLockMode(Method method) throws ExecutionException, InterruptedException {
> final K k1 = kvGenerator.generateKey(method, 1);
> final V v1 = kvGenerator.generateValue(method, 1);
> final V v2 = kvGenerator.generateValue(method, 2);
> RemoteCache<K, V> remoteCache = remoteCacheWithForceReturnValue();
> final TransactionManager tm = remoteCache.getTransactionManager();
> ExecutorService executorService = Executors.newFixedThreadPool(2);
> Future<?> f1 = executorService.submit(() -> {
> try {
> TestingUtil.withTx(tm, () -> {
> V returnValue = remoteCache.put(k1, v1);
> Thread.sleep(2000);
> return returnValue;
> });
> } catch (Exception e) {
> throw new IllegalStateException(e);
> }
> });
> Future<?> f2 = executorService.submit(() -> {
> try {
> TestingUtil.withTx(tm, () -> {
> Thread.sleep(500);
> V returnValue = remoteCache.put(k1, v2);
> return returnValue;
> });
> } catch (Exception e) {
> throw new IllegalStateException(e);
> }
> });
> Assert.assertNull(f1.get());
> Assert.assertNull(f2.get());
> }
> {code}
> The integration test is failing due:
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.testMultiThreadsPessimisticLockMode(TxFunctionalTest.java:473)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:196)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:208)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:635)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:816)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1124)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
> at org.testng.TestRunner.privateRun(TestRunner.java:774)
> at org.testng.TestRunner.run(TestRunner.java:624)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:359)
> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:354)
> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:312)
> at org.testng.SuiteRunner.run(SuiteRunner.java:261)
> at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
> at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
> at org.testng.TestNG.runSuitesSequentially(TestNG.java:1215)
> at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
> at org.testng.TestNG.run(TestNG.java:1048)
> at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
> at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
> Caused by: java.lang.IllegalStateException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.lambda$testMultiThreadsPessimisticLockMode$1(TxFunctionalTest.java:457)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
> at org.infinispan.test.TestingUtil.lambda$withTxCallable$11(TestingUtil.java:1529)
> at org.infinispan.test.TestingUtil.withTx(TestingUtil.java:1507)
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.lambda$testMultiThreadsPessimisticLockMode$1(TxFunctionalTest.java:451)
> ... 5 more
> Caused by: java.lang.Throwable: setRollbackOnly called from:
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.setRollbackOnly(TransactionImple.java:339)
> at org.infinispan.client.hotrod.impl.transaction.SyncModeTransactionTable$SynchronizationAdapter.markAsRollback(SyncModeTransactionTable.java:150)
> at org.infinispan.client.hotrod.impl.transaction.SyncModeTransactionTable$SynchronizationAdapter.beforeCompletion(SyncModeTransactionTable.java:125)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:371)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
> ... 9 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9420) HotRod Transaction Cache is throwing an exception with multiple threads
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-9420?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-9420:
-----------------------------------
No, it shouldn't.
Any transactional resources interact with the {{TransactionManager}} via xa-codes. In case of an exception, Infinispan throws a XaException(<xa-error-code>) back to the {{TransactionManager}}. The {{TransactionManager}} catches those exceptions and decides the next step. The user code (your test) will only see a {{RollbackException}} (as defined in the API).
Any cause exception ({{TimeoutException}} or {{WriteSkewException}}, for example) will be logged only. It won't reach the user code.
> HotRod Transaction Cache is throwing an exception with multiple threads
> -----------------------------------------------------------------------
>
> Key: ISPN-9420
> URL: https://issues.jboss.org/browse/ISPN-9420
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
>
> I created the method `public void testMultiThreadsPessimisticLockMode(Method method)` in the class `TxFunctionalTest`
> {code:java}
> public void testMultiThreadsPessimisticLockMode(Method method) throws ExecutionException, InterruptedException {
> final K k1 = kvGenerator.generateKey(method, 1);
> final V v1 = kvGenerator.generateValue(method, 1);
> final V v2 = kvGenerator.generateValue(method, 2);
> RemoteCache<K, V> remoteCache = remoteCacheWithForceReturnValue();
> final TransactionManager tm = remoteCache.getTransactionManager();
> ExecutorService executorService = Executors.newFixedThreadPool(2);
> Future<?> f1 = executorService.submit(() -> {
> try {
> TestingUtil.withTx(tm, () -> {
> V returnValue = remoteCache.put(k1, v1);
> Thread.sleep(2000);
> return returnValue;
> });
> } catch (Exception e) {
> throw new IllegalStateException(e);
> }
> });
> Future<?> f2 = executorService.submit(() -> {
> try {
> TestingUtil.withTx(tm, () -> {
> Thread.sleep(500);
> V returnValue = remoteCache.put(k1, v2);
> return returnValue;
> });
> } catch (Exception e) {
> throw new IllegalStateException(e);
> }
> });
> Assert.assertNull(f1.get());
> Assert.assertNull(f2.get());
> }
> {code}
> The integration test is failing due:
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.testMultiThreadsPessimisticLockMode(TxFunctionalTest.java:473)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:196)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:208)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:635)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:816)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1124)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
> at org.testng.TestRunner.privateRun(TestRunner.java:774)
> at org.testng.TestRunner.run(TestRunner.java:624)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:359)
> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:354)
> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:312)
> at org.testng.SuiteRunner.run(SuiteRunner.java:261)
> at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
> at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
> at org.testng.TestNG.runSuitesSequentially(TestNG.java:1215)
> at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
> at org.testng.TestNG.run(TestNG.java:1048)
> at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
> at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
> Caused by: java.lang.IllegalStateException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.lambda$testMultiThreadsPessimisticLockMode$1(TxFunctionalTest.java:457)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
> at org.infinispan.test.TestingUtil.lambda$withTxCallable$11(TestingUtil.java:1529)
> at org.infinispan.test.TestingUtil.withTx(TestingUtil.java:1507)
> at org.infinispan.client.hotrod.tx.TxFunctionalTest.lambda$testMultiThreadsPessimisticLockMode$1(TxFunctionalTest.java:451)
> ... 5 more
> Caused by: java.lang.Throwable: setRollbackOnly called from:
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.setRollbackOnly(TransactionImple.java:339)
> at org.infinispan.client.hotrod.impl.transaction.SyncModeTransactionTable$SynchronizationAdapter.markAsRollback(SyncModeTransactionTable.java:150)
> at org.infinispan.client.hotrod.impl.transaction.SyncModeTransactionTable$SynchronizationAdapter.beforeCompletion(SyncModeTransactionTable.java:125)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:371)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
> ... 9 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months