[JBoss JIRA] (ISPN-8997) Publish maven snapshot artifacts on stable master builds
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8997?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-8997:
----------------------------------
Sprint: Sprint 9.3.0.Alpha1, Sprint 9.3.0.Beta1, Sprint 9.3.0.CR1 (was: Sprint 9.3.0.Alpha1, Sprint 9.3.0.Beta1)
> Publish maven snapshot artifacts on stable master builds
> --------------------------------------------------------
>
> Key: ISPN-8997
> URL: https://issues.jboss.org/browse/ISPN-8997
> Project: Infinispan
> Issue Type: Bug
> Components: Build, CI
> Reporter: Tristan Tarrant
> Assignee: Sebastian Łaskawiec
>
> Successful builds of the master branch should publish snapshot artifacts to Maven
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-8731) Write command times out waiting for wrong topology
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8731?page=com.atlassian.jira.plugin.... ]
Dan Berindei edited comment on ISPN-8731 at 5/14/18 3:08 AM:
-------------------------------------------------------------
The failures happen because the {{DistributionManagerImpl}} topology is updated after the {{StateConsumerImpl}} topology (ISPN-7682).
{{BaseDistributionInterceptor.checkTopologyId()}} doesn't expect the command topology id (5) to be higher than the DM's current topology id, so it throws an {{OutdatedTopologyException}} without a topology id. {{StateTransferInterceptor.handleNonTxWriteCommand()}} then increments the command's topology id and times out waiting for topology 6.
{noformat}
09:47:27,497 TRACE (transport-thread-test-NodeA-p106-t6:[Topology-TestCache]) [StateConsumerImpl] Received new topology for cache TestCache, isRebalance = false, isMember = true, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=PartitionerConsistentHash:DefaultConsistentHash{ns=256, owners = (2)[test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1): 134+122, test-NodeB-59947 (flags=0), site-id=s2, rack-id=r2, machine-id=m2): 122+134]}, pendingCH=null, unionCH=null, actualMembers=[test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1), test-NodeB-59947 (flags=0), site-id=s2, rack-id=r2, machine-id=m2)], persistentUUIDs=[1ffbf082-035b-432d-8715-197ab9fbcab2, 7b2ba190-f559-4dc1-b874-5c43c3f4ce29]}
09:47:27,524 TRACE (testng-test:[]) [StateTransferInterceptor] handleNonTxWriteCommand for command PutKeyValueCommand{key=key1, value=value1, flags=[], commandInvocationId=CommandInvocation:test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1):36, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=-1}, topology id -1
09:47:27,524 TRACE (testng-test:[]) [StateTransferInterceptor] Setting command topology to 5
09:47:27,524 TRACE (testng-test:[]) [BaseDistributionInterceptor] Current topology 4, command topology 5
09:47:27,526 TRACE (testng-test:[]) [StateTransferInterceptor] Retrying command because of org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 5, got 4, current topology is 5 (requested: 6): PutKeyValueCommand{key=key1, value=value1, flags=[], commandInvocationId=CommandInvocation:test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1):36, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=5}
09:47:27,526 TRACE (testng-test:[]) [StateTransferLockImpl] Waiting for transaction data for topology 6, current topology is 4
09:47:27,545 TRACE (transport-thread-test-NodeA-p106-t6:[Topology-TestCache]) [StateTransferLockImpl] Signalling topology 5 is installed
09:47:42,549 ERROR (testng-test:[]) [TestSuiteProgress] Test failed: org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.testExecutionPolicyNotSameMachine
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1686) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1349) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1812) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:223) ~[classes/:?]
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[classes/:?]
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:673) ~[classes/:?]
at org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.executeDifferentExecutionPolicies(DistributedExecutorExecutionPolicyTest.java:115) ~[test-classes/:?]
at org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.testExecutionPolicyNotSameMachine(DistributedExecutorExecutionPolicyTest.java:59) ~[test-classes/:?]
{noformat}
was (Author: dan.berindei):
The failures happen because the {{DistributionManagerImpl}} topology is updated after the {{StateConsumerImpl}} topology (ISPN-7682).
{noformat}
09:47:27,497 TRACE (transport-thread-test-NodeA-p106-t6:[Topology-TestCache]) [StateConsumerImpl] Received new topology for cache TestCache, isRebalance = false, isMember = true, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=PartitionerConsistentHash:DefaultConsistentHash{ns=256, owners = (2)[test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1): 134+122, test-NodeB-59947 (flags=0), site-id=s2, rack-id=r2, machine-id=m2): 122+134]}, pendingCH=null, unionCH=null, actualMembers=[test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1), test-NodeB-59947 (flags=0), site-id=s2, rack-id=r2, machine-id=m2)], persistentUUIDs=[1ffbf082-035b-432d-8715-197ab9fbcab2, 7b2ba190-f559-4dc1-b874-5c43c3f4ce29]}
09:47:27,524 TRACE (testng-test:[]) [StateTransferInterceptor] handleNonTxWriteCommand for command PutKeyValueCommand{key=key1, value=value1, flags=[], commandInvocationId=CommandInvocation:test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1):36, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=-1}, topology id -1
09:47:27,524 TRACE (testng-test:[]) [StateTransferInterceptor] Setting command topology to 5
09:47:27,524 TRACE (testng-test:[]) [BaseDistributionInterceptor] Current topology 4, command topology 5
09:47:27,526 TRACE (testng-test:[]) [StateTransferInterceptor] Retrying command because of org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 5, got 4, current topology is 5 (requested: 6): PutKeyValueCommand{key=key1, value=value1, flags=[], commandInvocationId=CommandInvocation:test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1):36, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=5}
09:47:27,526 TRACE (testng-test:[]) [StateTransferLockImpl] Waiting for transaction data for topology 6, current topology is 4
09:47:27,545 TRACE (transport-thread-test-NodeA-p106-t6:[Topology-TestCache]) [StateTransferLockImpl] Signalling topology 5 is installed
09:47:42,549 ERROR (testng-test:[]) [TestSuiteProgress] Test failed: org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.testExecutionPolicyNotSameMachine
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1686) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1349) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1812) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:223) ~[classes/:?]
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[classes/:?]
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:673) ~[classes/:?]
at org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.executeDifferentExecutionPolicies(DistributedExecutorExecutionPolicyTest.java:115) ~[test-classes/:?]
at org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.testExecutionPolicyNotSameMachine(DistributedExecutorExecutionPolicyTest.java:59) ~[test-classes/:?]
{noformat}
> Write command times out waiting for wrong topology
> --------------------------------------------------
>
> Key: ISPN-8731
> URL: https://issues.jboss.org/browse/ISPN-8731
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Query, Test Suite - Server
> Affects Versions: 9.2.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.3.0.Final
>
>
> {{MultipleCacheManagersTest.waitForClusterToForm(new String[]{})}} doesn't actually wait for the rebalance to finish everywhere: https://github.com/infinispan/infinispan/pull/5705/files#diff-f9f535214b9...
> Most tests should work regardless of whether we wait for the rebalance to finish or not, it's just a way to reduce the number of test failures when the command retry doesn't work. Waiting for the rebalance to finish is only required when the test needs to install new topologies in a specific order.
> However, some recent test failures in CI show that the effectiveness of {{MultipleCacheManagersTest.waitForClusterToForm(new String[]{})}} does make a difference. Most likely there is a bug in the retry logic during the latter phases of a rebalance, and the retry-specific tests do not cover all the scenarios.
> {noformat}
> 00:23:20,668 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}, availability mode = AVAILABLE
> 00:23:20,669 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide stable topology for cache ___defaultcache, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:20,669 DEBUG (transport-thread-ClusteredScriptingTest-NodeI-p220-t6:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:20,669 DEBUG (transport-thread-ClusteredScriptingTest-NodeJ-p234-t6:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:35,673 ERROR (timeout-thread-ClusteredScriptingTest-NodeI-p219-t1:[]) [InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand, writing keys [/macbeth.txt0]
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:333) [infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> 00:23:35,673 DEBUG (testng-ClusteredScriptingTest:[]) [DefaultCacheManager] Stopping cache manager ISPN on ClusteredScriptingTest-NodeJ-9982
> 00:23:35,682 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=6, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[ClusteredScriptingTest-NodeI-40041: 256]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5]}, availability mode = AVAILABLE
> 00:23:35,699 ERROR (testng-ClusteredScriptingTest:[]) [TestSuiteProgress] Test failed: org.infinispan.scripting.ClusteredScriptingTest.testDistributedMapReduceStreamWithFlag([REPL_SYNC])
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1636) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1284) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1750) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:217) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:671) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.scripting.utils.ScriptingUtils.loadData(ScriptingUtils.java:29) ~[test-classes/:?]
> at org.infinispan.scripting.ClusteredScriptingTest$6.call(ClusteredScriptingTest.java:144) ~[test-classes/:?]
> at org.infinispan.test.TestingUtil.withCacheManagers(TestingUtil.java:1522) ~[infinispan-core-9.2.0-SNAPSHOT-tests.jar:9.2.0-SNAPSHOT]
> at org.infinispan.scripting.ClusteredScriptingTest.testDistributedMapReduceStreamWithFlag(ClusteredScriptingTest.java:137) ~[test-classes/:?]
> {noformat}
> {noformat}
> 18:04:17,097 DEBUG (transport-thread-ClusteredCacheTest-NodeB-p3689-t1:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredCacheTest-NodeA-53655: 134, ClusteredCacheTest-NodeB-27398: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655, ClusteredCacheTest-NodeB-27398], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362, b623ac7a-3c77-471d-88de-e151da823c0c]}
> 18:04:17,097 INFO (testng-ClusteredCacheTest:[]) [TestSuiteProgress] Test starting: org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner
> 18:04:17,097 DEBUG (transport-thread-ClusteredCacheTest-NodeA-p3679-t5:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredCacheTest-NodeA-53655: 134, ClusteredCacheTest-NodeB-27398: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655, ClusteredCacheTest-NodeB-27398], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362, b623ac7a-3c77-471d-88de-e151da823c0c]}
> 18:04:32,100 ERROR (timeout-thread-ClusteredCacheTest-NodeA-p3678-t1:[]) [InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand, writing keys [WrappedByteArray{bytes=[B0x010201054E617669..[9], hashCode=-1707624030}]
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:333) [infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> 18:04:32,101 ERROR (testng-ClusteredCacheTest:[]) [TestSuiteProgress] Test failed: org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1636) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1284) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1750) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:217) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:671) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.query.blackbox.ClusteredCacheTest.prepareTestData(ClusteredCacheTest.java:140) ~[test-classes/:?]
> at org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFrom(ClusteredCacheTest.java:334) ~[test-classes/:?]
> at org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner(ClusteredCacheTest.java:298) ~[test-classes/:?]
> 18:04:32,214 DEBUG (testng-ClusteredCacheTest:[]) [DefaultCacheManager] Stopping cache manager ISPN on ClusteredCacheTest-NodeB-27398
> 18:04:32,479 DEBUG (remote-thread-ClusteredCacheTest-NodeA-p3677-t6:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=6, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[ClusteredCacheTest-NodeA-53655: 256]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362]}, availability mode = AVAILABLE
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-8731) Write command times out waiting for wrong topology
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8731?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-8731:
------------------------------------
The failures happen because the {{DistributionManagerImpl}} topology is updated after the {{StateConsumerImpl}} topology (ISPN-7682).
{noformat}
09:47:27,497 TRACE (transport-thread-test-NodeA-p106-t6:[Topology-TestCache]) [StateConsumerImpl] Received new topology for cache TestCache, isRebalance = false, isMember = true, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=PartitionerConsistentHash:DefaultConsistentHash{ns=256, owners = (2)[test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1): 134+122, test-NodeB-59947 (flags=0), site-id=s2, rack-id=r2, machine-id=m2): 122+134]}, pendingCH=null, unionCH=null, actualMembers=[test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1), test-NodeB-59947 (flags=0), site-id=s2, rack-id=r2, machine-id=m2)], persistentUUIDs=[1ffbf082-035b-432d-8715-197ab9fbcab2, 7b2ba190-f559-4dc1-b874-5c43c3f4ce29]}
09:47:27,524 TRACE (testng-test:[]) [StateTransferInterceptor] handleNonTxWriteCommand for command PutKeyValueCommand{key=key1, value=value1, flags=[], commandInvocationId=CommandInvocation:test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1):36, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=-1}, topology id -1
09:47:27,524 TRACE (testng-test:[]) [StateTransferInterceptor] Setting command topology to 5
09:47:27,524 TRACE (testng-test:[]) [BaseDistributionInterceptor] Current topology 4, command topology 5
09:47:27,526 TRACE (testng-test:[]) [StateTransferInterceptor] Retrying command because of org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 5, got 4, current topology is 5 (requested: 6): PutKeyValueCommand{key=key1, value=value1, flags=[], commandInvocationId=CommandInvocation:test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1):36, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=5}
09:47:27,526 TRACE (testng-test:[]) [StateTransferLockImpl] Waiting for transaction data for topology 6, current topology is 4
09:47:27,545 TRACE (transport-thread-test-NodeA-p106-t6:[Topology-TestCache]) [StateTransferLockImpl] Signalling topology 5 is installed
09:47:42,549 ERROR (testng-test:[]) [TestSuiteProgress] Test failed: org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.testExecutionPolicyNotSameMachine
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1686) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1349) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1812) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:223) ~[classes/:?]
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[classes/:?]
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:673) ~[classes/:?]
at org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.executeDifferentExecutionPolicies(DistributedExecutorExecutionPolicyTest.java:115) ~[test-classes/:?]
at org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.testExecutionPolicyNotSameMachine(DistributedExecutorExecutionPolicyTest.java:59) ~[test-classes/:?]
{noformat}
> Write command times out waiting for wrong topology
> --------------------------------------------------
>
> Key: ISPN-8731
> URL: https://issues.jboss.org/browse/ISPN-8731
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Query, Test Suite - Server
> Affects Versions: 9.2.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.3.0.Final
>
>
> {{MultipleCacheManagersTest.waitForClusterToForm(new String[]{})}} doesn't actually wait for the rebalance to finish everywhere: https://github.com/infinispan/infinispan/pull/5705/files#diff-f9f535214b9...
> Most tests should work regardless of whether we wait for the rebalance to finish or not, it's just a way to reduce the number of test failures when the command retry doesn't work. Waiting for the rebalance to finish is only required when the test needs to install new topologies in a specific order.
> However, some recent test failures in CI show that the effectiveness of {{MultipleCacheManagersTest.waitForClusterToForm(new String[]{})}} does make a difference. Most likely there is a bug in the retry logic during the latter phases of a rebalance, and the retry-specific tests do not cover all the scenarios.
> {noformat}
> 00:23:20,668 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}, availability mode = AVAILABLE
> 00:23:20,669 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide stable topology for cache ___defaultcache, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:20,669 DEBUG (transport-thread-ClusteredScriptingTest-NodeI-p220-t6:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:20,669 DEBUG (transport-thread-ClusteredScriptingTest-NodeJ-p234-t6:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:35,673 ERROR (timeout-thread-ClusteredScriptingTest-NodeI-p219-t1:[]) [InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand, writing keys [/macbeth.txt0]
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:333) [infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> 00:23:35,673 DEBUG (testng-ClusteredScriptingTest:[]) [DefaultCacheManager] Stopping cache manager ISPN on ClusteredScriptingTest-NodeJ-9982
> 00:23:35,682 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=6, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[ClusteredScriptingTest-NodeI-40041: 256]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5]}, availability mode = AVAILABLE
> 00:23:35,699 ERROR (testng-ClusteredScriptingTest:[]) [TestSuiteProgress] Test failed: org.infinispan.scripting.ClusteredScriptingTest.testDistributedMapReduceStreamWithFlag([REPL_SYNC])
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1636) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1284) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1750) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:217) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:671) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.scripting.utils.ScriptingUtils.loadData(ScriptingUtils.java:29) ~[test-classes/:?]
> at org.infinispan.scripting.ClusteredScriptingTest$6.call(ClusteredScriptingTest.java:144) ~[test-classes/:?]
> at org.infinispan.test.TestingUtil.withCacheManagers(TestingUtil.java:1522) ~[infinispan-core-9.2.0-SNAPSHOT-tests.jar:9.2.0-SNAPSHOT]
> at org.infinispan.scripting.ClusteredScriptingTest.testDistributedMapReduceStreamWithFlag(ClusteredScriptingTest.java:137) ~[test-classes/:?]
> {noformat}
> {noformat}
> 18:04:17,097 DEBUG (transport-thread-ClusteredCacheTest-NodeB-p3689-t1:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredCacheTest-NodeA-53655: 134, ClusteredCacheTest-NodeB-27398: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655, ClusteredCacheTest-NodeB-27398], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362, b623ac7a-3c77-471d-88de-e151da823c0c]}
> 18:04:17,097 INFO (testng-ClusteredCacheTest:[]) [TestSuiteProgress] Test starting: org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner
> 18:04:17,097 DEBUG (transport-thread-ClusteredCacheTest-NodeA-p3679-t5:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredCacheTest-NodeA-53655: 134, ClusteredCacheTest-NodeB-27398: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655, ClusteredCacheTest-NodeB-27398], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362, b623ac7a-3c77-471d-88de-e151da823c0c]}
> 18:04:32,100 ERROR (timeout-thread-ClusteredCacheTest-NodeA-p3678-t1:[]) [InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand, writing keys [WrappedByteArray{bytes=[B0x010201054E617669..[9], hashCode=-1707624030}]
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:333) [infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> 18:04:32,101 ERROR (testng-ClusteredCacheTest:[]) [TestSuiteProgress] Test failed: org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1636) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1284) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1750) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:217) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:671) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.query.blackbox.ClusteredCacheTest.prepareTestData(ClusteredCacheTest.java:140) ~[test-classes/:?]
> at org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFrom(ClusteredCacheTest.java:334) ~[test-classes/:?]
> at org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner(ClusteredCacheTest.java:298) ~[test-classes/:?]
> 18:04:32,214 DEBUG (testng-ClusteredCacheTest:[]) [DefaultCacheManager] Stopping cache manager ISPN on ClusteredCacheTest-NodeB-27398
> 18:04:32,479 DEBUG (remote-thread-ClusteredCacheTest-NodeA-p3677-t6:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=6, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[ClusteredCacheTest-NodeA-53655: 256]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362]}, availability mode = AVAILABLE
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-9146) [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-9146?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated ISPN-9146:
---------------------------------
Affects Version/s: 9.2.2.Final
> [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9146
> URL: https://issues.jboss.org/browse/ISPN-9146
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.2.Final
> Reporter: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> Important configuration change on JDG: default cache has transactions configured
> {code:xml}
> <distributed-cache name="default">
> <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
> </distributed-cache>
> {code}
> With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws
> {code}
> [JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
> [JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
> [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
> [JBossINF] at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
> [JBossINF] at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
> [JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> [JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> [JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
> [JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
> [JBossINF] ... 50 more
> {code}
> JDG log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> This results in EAP logging
> {code}
> [JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
> [JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace
> [JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] java.nio.channels.OverlappingFileLockException
> [JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> [JBossINF] at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> [JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76)
> [JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
> [JBossINF] at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
> [JBossINF] ... 7 more
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> [JBossINF] ... 31 more
> {code}
> EAP log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-9146) [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-9146?page=com.atlassian.jira.plugin.... ]
Radoslav Husar moved WFWIP-38 to ISPN-9146:
-------------------------------------------
Project: Infinispan (was: WildFly WIP)
Key: ISPN-9146 (was: WFWIP-38)
Workflow: GIT Pull Request with Triage workflow (was: GIT Pull Request workflow )
Component/s: Core
(was: Clustering)
> [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9146
> URL: https://issues.jboss.org/browse/ISPN-9146
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> Important configuration change on JDG: default cache has transactions configured
> {code:xml}
> <distributed-cache name="default">
> <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
> </distributed-cache>
> {code}
> With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws
> {code}
> [JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
> [JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
> [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
> [JBossINF] at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
> [JBossINF] at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
> [JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> [JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> [JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
> [JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
> [JBossINF] ... 50 more
> {code}
> JDG log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> This results in EAP logging
> {code}
> [JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
> [JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace
> [JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] java.nio.channels.OverlappingFileLockException
> [JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> [JBossINF] at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> [JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76)
> [JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
> [JBossINF] at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
> [JBossINF] ... 7 more
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> [JBossINF] ... 31 more
> {code}
> EAP log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-9146) [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-9146?page=com.atlassian.jira.plugin.... ]
Radoslav Husar reassigned ISPN-9146:
------------------------------------
Assignee: (was: Radoslav Husar)
> [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9146
> URL: https://issues.jboss.org/browse/ISPN-9146
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> Important configuration change on JDG: default cache has transactions configured
> {code:xml}
> <distributed-cache name="default">
> <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
> </distributed-cache>
> {code}
> With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws
> {code}
> [JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
> [JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
> [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
> [JBossINF] at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
> [JBossINF] at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
> [JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> [JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> [JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
> [JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
> [JBossINF] ... 50 more
> {code}
> JDG log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> This results in EAP logging
> {code}
> [JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
> [JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace
> [JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] java.nio.channels.OverlappingFileLockException
> [JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> [JBossINF] at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> [JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76)
> [JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
> [JBossINF] at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
> [JBossINF] ... 7 more
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> [JBossINF] ... 31 more
> {code}
> EAP log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-9144) Document AdvancedCacheLoader semantics
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-9144?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated ISPN-9144:
---------------------------------
Description:
Unfortunately the behavior of how stores operate is not very transparent to the user. For example you can configure multiple stores/loaders. However it is not clear how this works with bulk operations, such as size or iterator.
This was discussed in https://github.com/infinispan/infinispan/pull/5794#discussion_r173854664
was:
Unfortunately the behavior of how are stores operate is not very transparent to the user. For example you can configure multiple stores/loaders. However it is not clear how this works with bulk operations, such as size or iterator.
This was discussed in https://github.com/infinispan/infinispan/pull/5794#discussion_r173854664
> Document AdvancedCacheLoader semantics
> --------------------------------------
>
> Key: ISPN-9144
> URL: https://issues.jboss.org/browse/ISPN-9144
> Project: Infinispan
> Issue Type: Task
> Components: Loaders and Stores
> Reporter: William Burns
> Assignee: William Burns
>
> Unfortunately the behavior of how stores operate is not very transparent to the user. For example you can configure multiple stores/loaders. However it is not clear how this works with bulk operations, such as size or iterator.
> This was discussed in https://github.com/infinispan/infinispan/pull/5794#discussion_r173854664
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-8008) Add Fault-tolerance to write-behind stores
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-8008?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-8008:
--------------------------------------
Fix Version/s: 9.3.0.CR1
(was: 9.3.0.Beta1)
> Add Fault-tolerance to write-behind stores
> ------------------------------------------
>
> Key: ISPN-8008
> URL: https://issues.jboss.org/browse/ISPN-8008
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.3.0.CR1
>
>
> Currently when a store is configured to be write-behind, three attempts are made to write the queued modifications to the store. If all three attempts fail, then this error is simply logged and the modifications are never written to the store.
> We should allow users to configure a write-behind store so that: In the event that a write-behind fails, further operations on the cache are not allowed and the modifications which failed to write to the store are queued indefinitely. Once the underlying store becomes available, the queued modifications should be written and then the cache should become available.
> To determine whether a store has become available again, the AsyncCacheWriter will have to poll the store's availability. Currently the CacheWriter interface does not allow such behaviour, therefore there are two options:
> * Attempt a write and catch an exception if the store is not currently available
> * Add a method to return the current status of a store to the CacheWriter interface
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-7022) TwoWaySplitAndMergeTest.testSplitAndMerge2 fails
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-7022?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-7022:
--------------------------------------
Fix Version/s: 9.3.0.CR1
(was: 9.3.0.Beta1)
> TwoWaySplitAndMergeTest.testSplitAndMerge2 fails
> ------------------------------------------------
>
> Key: ISPN-7022
> URL: https://issues.jboss.org/browse/ISPN-7022
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.1.Final
> Reporter: Sebastian Łaskawiec
> Assignee: Dan Berindei
> Fix For: 9.3.0.CR1
>
>
> http://ci.infinispan.org/viewLog.html?buildId=42202&tab=buildResultsDiv&b...
> {noformat}
> java.lang.AssertionError:
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:235)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:217)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:193)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:527)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest.access$600(BasePartitionHandlingTest.java:44)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.assertAvailabilityMode(BasePartitionHandlingTest.java:328)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.assertDegradedMode(BasePartitionHandlingTest.java:265)
> at org.infinispan.partitionhandling.TwoWaySplitAndMergeTest.testSplitAndMerge(TwoWaySplitAndMergeTest.java:74)
> at org.infinispan.partitionhandling.TwoWaySplitAndMergeTest.testSplitAndMerge2(TwoWaySplitAndMergeTest.java:26)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> ------- Stdout: -------
> 13:26:54,221 ERROR (testng-DelayedAvailabilityUpdateTest) [InvocationContextInterceptor] ISPN000136: Error executing command GetKeyValueCommand, writing keys []
> org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'MagicKey#k1Existing{1420/A1FBD015/225@DelayedAvailabilityUpdateTest-NodeT-8452}' is not available. Not all owners are in this partition
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:250)
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkRead(PartitionHandlingManagerImpl.java:102)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.postOperationPartitionCheck(PartitionHandlingInterceptor.java:188)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.lambda$handleDataReadCommand$412(PartitionHandlingInterceptor.java:145)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor$$Lambda$1639/1963537113.handle(Unknown Source)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlerSync(BaseAsyncInvocationContext.java:493)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlersSync(BaseAsyncInvocationContext.java:125)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.forkInvocationSync(BaseAsyncInvocationContext.java:99)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeForkAndHandlerSync(BaseAsyncInvocationContext.java:474)[TestSuiteProgress] Test failed: org.infinispan.partitionhandling.OptimisticTxPartitionAndMergeDuringCommitTest.testOriginatorIsolatedPartition
> 13:26:54,221 ERROR (testng-OptimisticTxPartitionAndMergeDuringCommitTest) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.OptimisticTxPartitionAndMergeDuringCommitTest.testOriginatorIsolatedPartition
> java.lang.AssertionError: There are pending transactions!
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:235)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:531)
> at org.infinispan.test.MultipleCacheManagersTest.assertNoTransactions(MultipleCacheManagersTest.java:794)
> at org.infinispan.partitionhandling.BaseTxPartitionAndMergeTest.finalAsserts(BaseTxPartitionAndMergeTest.java:96)
> at org.infinispan.partitionhandling.BaseOptimisticTxPartitionAndMergeTest.doTest(BaseOptimisticTxPartitionAndMergeTest.java:77)
> at org.infinispan.partitionhandling.OptimisticTxPartitionAndMergeDuringCommitTest.testOriginatorIsolatedPartition(OptimisticTxPartitionAndMergeDuringCommitTest.java:37)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[TestSuiteProgress] Tests succeeded: 6704, failed: 1, skipped: 0
> 13:26:54,222 ERROR (testng-DelayedAvailabilityUpdateTest) [InvocationContextInterceptor] ISPN000136: Error executing command GetKeyValueCommand, writing keys []
> org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'MagicKey#k2Existing{1421/89F8758B/114@DelayedAvailabilityUpdateTest-NodeQ-17336}' is not available. Not all owners are in this partition
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.postOperationPartitionCheck(PartitionHandlingInterceptor.java:197)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.lambda$handleDataReadCommand$412(PartitionHandlingInterceptor.java:145)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor$$Lambda$1639/1963537113.handle(Unknown Source)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlerSync(BaseAsyncInvocationContext.java:493)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlersSync(BaseAsyncInvocationContext.java:125)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.forkInvocationSync(BaseAsyncInvocationContext.java:99)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeForkAndHandlerSync(BaseAsyncInvocationContext.java:474)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.afterVisit(BaseAsyncInvocationContext.java:463)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeInterceptorsSync(BaseAsyncInvocationContext.java:322)13:26:54,222 ERROR (testng-DelayedAvailabilityUpdateTest) [InvocationContextInterceptor] ISPN000136: Error executing command GetKeyValueCommand, writing keys []
> org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'MagicKey#k2Existing{1421/89F8758B/114@DelayedAvailabilityUpdateTest-NodeQ-17336}' is not available. Not all owners are in this partition
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:250)
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkRead(PartitionHandlingManagerImpl.java:102)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.postOperationPartitionCheck(PartitionHandlingInterceptor.java:188)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.lambda$handleDataReadCommand$412(PartitionHandlingInterceptor.java:145)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor$$Lambda$1639/1963537113.handle(Unknown Source)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlerSync(BaseAsyncInvocationContext.java:493)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlersSync(BaseAsyncInvocationContext.java:125)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.forkInvocationSync(BaseAsyncInvocationContext.java:99)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeForkAndHandlerSync(BaseAsyncInvocationContext.java:474)13:26:49,964 ERROR (transport-thread-PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-p44001-t2) [CLUSTER] [Context=pes-cache]ISPN000319: After merge (or coordinator change), cache still hasn't recovered all its data and must stay in degraded mode. Current members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-37404, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeH-18972], lost members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeE-27252, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeF-47835], stable members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeE-27252, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeF-47835, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-37404, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeH-18972]
> [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.TwoWaySplitAndMergeTest.testSplitAndMerge2
> 13:26:55,721 ERROR (testng-TwoWaySplitAndMergeTest) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.TwoWaySplitAndMergeTest.testSplitAndMerge2
> java.lang.AssertionError:
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:235)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:217)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:193)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:527)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest.access$600(BasePartitionHandlingTest.java:44)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.assertAvailabilityMode(BasePartitionHandlingTest.java:328)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.assertDegradedMode(BasePartitionHandlingTest.java:265)13:26:49,964 ERROR (transport-thread-PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-p44001-t3) [CLUSTER] [Context=___defaultcache]ISPN000319: After merge (or coordinator change), cache still hasn't recovered all its data and must stay in degraded mode. Current members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-37404, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeH-18972], lost members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeE-27252, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeF-47835], stable members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeE-27252, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeF-47835, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-37404, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeH-18972]
> [TestSuiteProgress] Tests succeeded: 6704, failed: 2, skipped: 0
> 13:26:55,095 ERROR (testng-OptimisticTxPartitionAndMergeDuringPrepareTest) [InvocationContextInterceptor] ISPN000136: Error executing command ClearCommand, writing keys []
> org.infinispan.partitionhandling.AvailabilityException: ISPN000307: Cannot clear when the cluster is partitioned
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkClear(PartitionHandlingManagerImpl.java:108)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitClearCommand(PartitionHandlingInterceptor.java:95)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitClearCommand(PartitionHandlingInterceptor.java:38)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:44)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:53)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeInterceptorsSync(BaseAsyncInvocationContext.java:307)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.forkInvocationSync(BaseAsyncInvocationContext.java:98)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeForkAndHandlerSync(BaseAsyncInvocationContext.java:474)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.afterVisit(BaseAsyncInvocationContext.java:463)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-9029) Add test for exception eviction dropping entries after node leave
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-9029?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-9029:
--------------------------------------
Fix Version/s: 9.3.0.CR1
(was: 9.3.0.Beta1)
> Add test for exception eviction dropping entries after node leave
> -----------------------------------------------------------------
>
> Key: ISPN-9029
> URL: https://issues.jboss.org/browse/ISPN-9029
> Project: Infinispan
> Issue Type: Task
> Components: Test Suite - Core
> Affects Versions: 9.2.1.Final
> Reporter: Dan Berindei
> Assignee: William Burns
> Fix For: 9.3.0.CR1
>
>
> {{ExceptionEvictionTest}} tests what happens when a node is added and N*limit existing entries are redistributed among N+1 nodes. We also need to test what happens when a node crashes and N*limit entries are redistributed among N-1 nodes.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0