[JBoss JIRA] (ISPN-8731) Write command times out waiting for wrong topology
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8731?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-8731:
------------------------------------
The failures happen because the {{DistributionManagerImpl}} topology is updated after the {{StateConsumerImpl}} topology (ISPN-7682).
{noformat}
09:47:27,497 TRACE (transport-thread-test-NodeA-p106-t6:[Topology-TestCache]) [StateConsumerImpl] Received new topology for cache TestCache, isRebalance = false, isMember = true, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=PartitionerConsistentHash:DefaultConsistentHash{ns=256, owners = (2)[test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1): 134+122, test-NodeB-59947 (flags=0), site-id=s2, rack-id=r2, machine-id=m2): 122+134]}, pendingCH=null, unionCH=null, actualMembers=[test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1), test-NodeB-59947 (flags=0), site-id=s2, rack-id=r2, machine-id=m2)], persistentUUIDs=[1ffbf082-035b-432d-8715-197ab9fbcab2, 7b2ba190-f559-4dc1-b874-5c43c3f4ce29]}
09:47:27,524 TRACE (testng-test:[]) [StateTransferInterceptor] handleNonTxWriteCommand for command PutKeyValueCommand{key=key1, value=value1, flags=[], commandInvocationId=CommandInvocation:test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1):36, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=-1}, topology id -1
09:47:27,524 TRACE (testng-test:[]) [StateTransferInterceptor] Setting command topology to 5
09:47:27,524 TRACE (testng-test:[]) [BaseDistributionInterceptor] Current topology 4, command topology 5
09:47:27,526 TRACE (testng-test:[]) [StateTransferInterceptor] Retrying command because of org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 5, got 4, current topology is 5 (requested: 6): PutKeyValueCommand{key=key1, value=value1, flags=[], commandInvocationId=CommandInvocation:test-NodeA-14300 (flags=0), site-id=s1, rack-id=r1, machine-id=m1):36, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=5}
09:47:27,526 TRACE (testng-test:[]) [StateTransferLockImpl] Waiting for transaction data for topology 6, current topology is 4
09:47:27,545 TRACE (transport-thread-test-NodeA-p106-t6:[Topology-TestCache]) [StateTransferLockImpl] Signalling topology 5 is installed
09:47:42,549 ERROR (testng-test:[]) [TestSuiteProgress] Test failed: org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.testExecutionPolicyNotSameMachine
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1686) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1349) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1812) ~[classes/:?]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:223) ~[classes/:?]
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[classes/:?]
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:673) ~[classes/:?]
at org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.executeDifferentExecutionPolicies(DistributedExecutorExecutionPolicyTest.java:115) ~[test-classes/:?]
at org.infinispan.distexec.DistributedExecutorExecutionPolicyTest.testExecutionPolicyNotSameMachine(DistributedExecutorExecutionPolicyTest.java:59) ~[test-classes/:?]
{noformat}
> Write command times out waiting for wrong topology
> --------------------------------------------------
>
> Key: ISPN-8731
> URL: https://issues.jboss.org/browse/ISPN-8731
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Query, Test Suite - Server
> Affects Versions: 9.2.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.3.0.Final
>
>
> {{MultipleCacheManagersTest.waitForClusterToForm(new String[]{})}} doesn't actually wait for the rebalance to finish everywhere: https://github.com/infinispan/infinispan/pull/5705/files#diff-f9f535214b9...
> Most tests should work regardless of whether we wait for the rebalance to finish or not, it's just a way to reduce the number of test failures when the command retry doesn't work. Waiting for the rebalance to finish is only required when the test needs to install new topologies in a specific order.
> However, some recent test failures in CI show that the effectiveness of {{MultipleCacheManagersTest.waitForClusterToForm(new String[]{})}} does make a difference. Most likely there is a bug in the retry logic during the latter phases of a rebalance, and the retry-specific tests do not cover all the scenarios.
> {noformat}
> 00:23:20,668 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}, availability mode = AVAILABLE
> 00:23:20,669 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide stable topology for cache ___defaultcache, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:20,669 DEBUG (transport-thread-ClusteredScriptingTest-NodeI-p220-t6:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:20,669 DEBUG (transport-thread-ClusteredScriptingTest-NodeJ-p234-t6:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:35,673 ERROR (timeout-thread-ClusteredScriptingTest-NodeI-p219-t1:[]) [InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand, writing keys [/macbeth.txt0]
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:333) [infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> 00:23:35,673 DEBUG (testng-ClusteredScriptingTest:[]) [DefaultCacheManager] Stopping cache manager ISPN on ClusteredScriptingTest-NodeJ-9982
> 00:23:35,682 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=6, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[ClusteredScriptingTest-NodeI-40041: 256]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5]}, availability mode = AVAILABLE
> 00:23:35,699 ERROR (testng-ClusteredScriptingTest:[]) [TestSuiteProgress] Test failed: org.infinispan.scripting.ClusteredScriptingTest.testDistributedMapReduceStreamWithFlag([REPL_SYNC])
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1636) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1284) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1750) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:217) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:671) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.scripting.utils.ScriptingUtils.loadData(ScriptingUtils.java:29) ~[test-classes/:?]
> at org.infinispan.scripting.ClusteredScriptingTest$6.call(ClusteredScriptingTest.java:144) ~[test-classes/:?]
> at org.infinispan.test.TestingUtil.withCacheManagers(TestingUtil.java:1522) ~[infinispan-core-9.2.0-SNAPSHOT-tests.jar:9.2.0-SNAPSHOT]
> at org.infinispan.scripting.ClusteredScriptingTest.testDistributedMapReduceStreamWithFlag(ClusteredScriptingTest.java:137) ~[test-classes/:?]
> {noformat}
> {noformat}
> 18:04:17,097 DEBUG (transport-thread-ClusteredCacheTest-NodeB-p3689-t1:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredCacheTest-NodeA-53655: 134, ClusteredCacheTest-NodeB-27398: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655, ClusteredCacheTest-NodeB-27398], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362, b623ac7a-3c77-471d-88de-e151da823c0c]}
> 18:04:17,097 INFO (testng-ClusteredCacheTest:[]) [TestSuiteProgress] Test starting: org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner
> 18:04:17,097 DEBUG (transport-thread-ClusteredCacheTest-NodeA-p3679-t5:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredCacheTest-NodeA-53655: 134, ClusteredCacheTest-NodeB-27398: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655, ClusteredCacheTest-NodeB-27398], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362, b623ac7a-3c77-471d-88de-e151da823c0c]}
> 18:04:32,100 ERROR (timeout-thread-ClusteredCacheTest-NodeA-p3678-t1:[]) [InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand, writing keys [WrappedByteArray{bytes=[B0x010201054E617669..[9], hashCode=-1707624030}]
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:333) [infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> 18:04:32,101 ERROR (testng-ClusteredCacheTest:[]) [TestSuiteProgress] Test failed: org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1636) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1284) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1750) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:217) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:671) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.query.blackbox.ClusteredCacheTest.prepareTestData(ClusteredCacheTest.java:140) ~[test-classes/:?]
> at org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFrom(ClusteredCacheTest.java:334) ~[test-classes/:?]
> at org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner(ClusteredCacheTest.java:298) ~[test-classes/:?]
> 18:04:32,214 DEBUG (testng-ClusteredCacheTest:[]) [DefaultCacheManager] Stopping cache manager ISPN on ClusteredCacheTest-NodeB-27398
> 18:04:32,479 DEBUG (remote-thread-ClusteredCacheTest-NodeA-p3677-t6:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=6, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[ClusteredCacheTest-NodeA-53655: 256]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362]}, availability mode = AVAILABLE
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
[JBoss JIRA] (ISPN-9146) [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-9146?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated ISPN-9146:
---------------------------------
Affects Version/s: 9.2.2.Final
> [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9146
> URL: https://issues.jboss.org/browse/ISPN-9146
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.2.Final
> Reporter: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> Important configuration change on JDG: default cache has transactions configured
> {code:xml}
> <distributed-cache name="default">
> <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
> </distributed-cache>
> {code}
> With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws
> {code}
> [JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
> [JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
> [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
> [JBossINF] at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
> [JBossINF] at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
> [JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> [JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> [JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
> [JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
> [JBossINF] ... 50 more
> {code}
> JDG log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> This results in EAP logging
> {code}
> [JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
> [JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace
> [JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] java.nio.channels.OverlappingFileLockException
> [JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> [JBossINF] at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> [JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76)
> [JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
> [JBossINF] at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
> [JBossINF] ... 7 more
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> [JBossINF] ... 31 more
> {code}
> EAP log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-9146) [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-9146?page=com.atlassian.jira.plugin.... ]
Radoslav Husar moved WFWIP-38 to ISPN-9146:
-------------------------------------------
Project: Infinispan (was: WildFly WIP)
Key: ISPN-9146 (was: WFWIP-38)
Workflow: GIT Pull Request with Triage workflow (was: GIT Pull Request workflow )
Component/s: Core
(was: Clustering)
> [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9146
> URL: https://issues.jboss.org/browse/ISPN-9146
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> Important configuration change on JDG: default cache has transactions configured
> {code:xml}
> <distributed-cache name="default">
> <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
> </distributed-cache>
> {code}
> With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws
> {code}
> [JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
> [JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
> [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
> [JBossINF] at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
> [JBossINF] at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
> [JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> [JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> [JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
> [JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
> [JBossINF] ... 50 more
> {code}
> JDG log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> This results in EAP logging
> {code}
> [JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
> [JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace
> [JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] java.nio.channels.OverlappingFileLockException
> [JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> [JBossINF] at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> [JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76)
> [JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
> [JBossINF] at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
> [JBossINF] ... 7 more
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> [JBossINF] ... 31 more
> {code}
> EAP log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-9146) [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-9146?page=com.atlassian.jira.plugin.... ]
Radoslav Husar reassigned ISPN-9146:
------------------------------------
Assignee: (was: Radoslav Husar)
> [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9146
> URL: https://issues.jboss.org/browse/ISPN-9146
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> Important configuration change on JDG: default cache has transactions configured
> {code:xml}
> <distributed-cache name="default">
> <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
> </distributed-cache>
> {code}
> With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws
> {code}
> [JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
> [JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
> [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
> [JBossINF] at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
> [JBossINF] at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
> [JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> [JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> [JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
> [JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
> [JBossINF] ... 50 more
> {code}
> JDG log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> This results in EAP logging
> {code}
> [JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
> [JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace
> [JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
> [JBossINF] java.nio.channels.OverlappingFileLockException
> [JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> [JBossINF] at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> [JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76)
> [JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
> [JBossINF] at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
> [JBossINF] ... 7 more
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> [JBossINF] ... 31 more
> {code}
> EAP log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8008) Add Fault-tolerance to write-behind stores
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-8008?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-8008:
--------------------------------------
Fix Version/s: 9.3.0.CR1
(was: 9.3.0.Beta1)
> Add Fault-tolerance to write-behind stores
> ------------------------------------------
>
> Key: ISPN-8008
> URL: https://issues.jboss.org/browse/ISPN-8008
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.3.0.CR1
>
>
> Currently when a store is configured to be write-behind, three attempts are made to write the queued modifications to the store. If all three attempts fail, then this error is simply logged and the modifications are never written to the store.
> We should allow users to configure a write-behind store so that: In the event that a write-behind fails, further operations on the cache are not allowed and the modifications which failed to write to the store are queued indefinitely. Once the underlying store becomes available, the queued modifications should be written and then the cache should become available.
> To determine whether a store has become available again, the AsyncCacheWriter will have to poll the store's availability. Currently the CacheWriter interface does not allow such behaviour, therefore there are two options:
> * Attempt a write and catch an exception if the store is not currently available
> * Add a method to return the current status of a store to the CacheWriter interface
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-7022) TwoWaySplitAndMergeTest.testSplitAndMerge2 fails
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-7022?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-7022:
--------------------------------------
Fix Version/s: 9.3.0.CR1
(was: 9.3.0.Beta1)
> TwoWaySplitAndMergeTest.testSplitAndMerge2 fails
> ------------------------------------------------
>
> Key: ISPN-7022
> URL: https://issues.jboss.org/browse/ISPN-7022
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.1.Final
> Reporter: Sebastian Łaskawiec
> Assignee: Dan Berindei
> Fix For: 9.3.0.CR1
>
>
> http://ci.infinispan.org/viewLog.html?buildId=42202&tab=buildResultsDiv&b...
> {noformat}
> java.lang.AssertionError:
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:235)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:217)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:193)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:527)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest.access$600(BasePartitionHandlingTest.java:44)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.assertAvailabilityMode(BasePartitionHandlingTest.java:328)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.assertDegradedMode(BasePartitionHandlingTest.java:265)
> at org.infinispan.partitionhandling.TwoWaySplitAndMergeTest.testSplitAndMerge(TwoWaySplitAndMergeTest.java:74)
> at org.infinispan.partitionhandling.TwoWaySplitAndMergeTest.testSplitAndMerge2(TwoWaySplitAndMergeTest.java:26)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> ------- Stdout: -------
> 13:26:54,221 ERROR (testng-DelayedAvailabilityUpdateTest) [InvocationContextInterceptor] ISPN000136: Error executing command GetKeyValueCommand, writing keys []
> org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'MagicKey#k1Existing{1420/A1FBD015/225@DelayedAvailabilityUpdateTest-NodeT-8452}' is not available. Not all owners are in this partition
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:250)
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkRead(PartitionHandlingManagerImpl.java:102)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.postOperationPartitionCheck(PartitionHandlingInterceptor.java:188)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.lambda$handleDataReadCommand$412(PartitionHandlingInterceptor.java:145)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor$$Lambda$1639/1963537113.handle(Unknown Source)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlerSync(BaseAsyncInvocationContext.java:493)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlersSync(BaseAsyncInvocationContext.java:125)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.forkInvocationSync(BaseAsyncInvocationContext.java:99)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeForkAndHandlerSync(BaseAsyncInvocationContext.java:474)[TestSuiteProgress] Test failed: org.infinispan.partitionhandling.OptimisticTxPartitionAndMergeDuringCommitTest.testOriginatorIsolatedPartition
> 13:26:54,221 ERROR (testng-OptimisticTxPartitionAndMergeDuringCommitTest) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.OptimisticTxPartitionAndMergeDuringCommitTest.testOriginatorIsolatedPartition
> java.lang.AssertionError: There are pending transactions!
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:235)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:531)
> at org.infinispan.test.MultipleCacheManagersTest.assertNoTransactions(MultipleCacheManagersTest.java:794)
> at org.infinispan.partitionhandling.BaseTxPartitionAndMergeTest.finalAsserts(BaseTxPartitionAndMergeTest.java:96)
> at org.infinispan.partitionhandling.BaseOptimisticTxPartitionAndMergeTest.doTest(BaseOptimisticTxPartitionAndMergeTest.java:77)
> at org.infinispan.partitionhandling.OptimisticTxPartitionAndMergeDuringCommitTest.testOriginatorIsolatedPartition(OptimisticTxPartitionAndMergeDuringCommitTest.java:37)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[TestSuiteProgress] Tests succeeded: 6704, failed: 1, skipped: 0
> 13:26:54,222 ERROR (testng-DelayedAvailabilityUpdateTest) [InvocationContextInterceptor] ISPN000136: Error executing command GetKeyValueCommand, writing keys []
> org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'MagicKey#k2Existing{1421/89F8758B/114@DelayedAvailabilityUpdateTest-NodeQ-17336}' is not available. Not all owners are in this partition
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.postOperationPartitionCheck(PartitionHandlingInterceptor.java:197)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.lambda$handleDataReadCommand$412(PartitionHandlingInterceptor.java:145)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor$$Lambda$1639/1963537113.handle(Unknown Source)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlerSync(BaseAsyncInvocationContext.java:493)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlersSync(BaseAsyncInvocationContext.java:125)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.forkInvocationSync(BaseAsyncInvocationContext.java:99)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeForkAndHandlerSync(BaseAsyncInvocationContext.java:474)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.afterVisit(BaseAsyncInvocationContext.java:463)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeInterceptorsSync(BaseAsyncInvocationContext.java:322)13:26:54,222 ERROR (testng-DelayedAvailabilityUpdateTest) [InvocationContextInterceptor] ISPN000136: Error executing command GetKeyValueCommand, writing keys []
> org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'MagicKey#k2Existing{1421/89F8758B/114@DelayedAvailabilityUpdateTest-NodeQ-17336}' is not available. Not all owners are in this partition
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:250)
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkRead(PartitionHandlingManagerImpl.java:102)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.postOperationPartitionCheck(PartitionHandlingInterceptor.java:188)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.lambda$handleDataReadCommand$412(PartitionHandlingInterceptor.java:145)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor$$Lambda$1639/1963537113.handle(Unknown Source)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlerSync(BaseAsyncInvocationContext.java:493)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeReturnHandlersSync(BaseAsyncInvocationContext.java:125)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.forkInvocationSync(BaseAsyncInvocationContext.java:99)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeForkAndHandlerSync(BaseAsyncInvocationContext.java:474)13:26:49,964 ERROR (transport-thread-PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-p44001-t2) [CLUSTER] [Context=pes-cache]ISPN000319: After merge (or coordinator change), cache still hasn't recovered all its data and must stay in degraded mode. Current members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-37404, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeH-18972], lost members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeE-27252, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeF-47835], stable members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeE-27252, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeF-47835, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-37404, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeH-18972]
> [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.TwoWaySplitAndMergeTest.testSplitAndMerge2
> 13:26:55,721 ERROR (testng-TwoWaySplitAndMergeTest) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.TwoWaySplitAndMergeTest.testSplitAndMerge2
> java.lang.AssertionError:
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:235)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:217)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:193)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:527)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest.access$600(BasePartitionHandlingTest.java:44)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.assertAvailabilityMode(BasePartitionHandlingTest.java:328)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.assertDegradedMode(BasePartitionHandlingTest.java:265)13:26:49,964 ERROR (transport-thread-PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-p44001-t3) [CLUSTER] [Context=___defaultcache]ISPN000319: After merge (or coordinator change), cache still hasn't recovered all its data and must stay in degraded mode. Current members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-37404, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeH-18972], lost members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeE-27252, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeF-47835], stable members are [PessimisticTxPartitionAndMergeDuringPrepareTest-NodeE-27252, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeF-47835, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeG-37404, PessimisticTxPartitionAndMergeDuringPrepareTest-NodeH-18972]
> [TestSuiteProgress] Tests succeeded: 6704, failed: 2, skipped: 0
> 13:26:55,095 ERROR (testng-OptimisticTxPartitionAndMergeDuringPrepareTest) [InvocationContextInterceptor] ISPN000136: Error executing command ClearCommand, writing keys []
> org.infinispan.partitionhandling.AvailabilityException: ISPN000307: Cannot clear when the cluster is partitioned
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkClear(PartitionHandlingManagerImpl.java:108)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitClearCommand(PartitionHandlingInterceptor.java:95)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitClearCommand(PartitionHandlingInterceptor.java:38)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:44)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:53)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeInterceptorsSync(BaseAsyncInvocationContext.java:307)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.forkInvocationSync(BaseAsyncInvocationContext.java:98)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.invokeForkAndHandlerSync(BaseAsyncInvocationContext.java:474)
> at org.infinispan.interceptors.impl.BaseAsyncInvocationContext.afterVisit(BaseAsyncInvocationContext.java:463)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-9028) Write-only segments should be invalidated during the READ_NEW phase
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-9028?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-9028:
--------------------------------------
Fix Version/s: 9.3.0.CR1
(was: 9.3.0.Beta1)
> Write-only segments should be invalidated during the READ_NEW phase
> -------------------------------------------------------------------
>
> Key: ISPN-9028
> URL: https://issues.jboss.org/browse/ISPN-9028
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.3.0.CR1
>
>
> When a rebalance removes a segment X from node A, node A keeps updating entries in segment X until the rebalance finishes, and only deletes the entries of segment X after entering the NO_REBALANCE phase.
> This is problematic for tests that work with the data container directly, because {{waitForNoRebalance()}} doesn't wait for the removal of stale entries. The test will work without an explicit wait most of the time, so this is a recipe for random test failures (e.g. ISPN-8728).
> As described in ISPN-5021, we can prevent any writes to segment X at the start of the READ_NEW_WRITE_ALL phase, send the phase confirmation to the coordinator, and then remove the entries asynchronously. We just need to keep track of the removal task and only install/confirm the NO_REBALANCE phase once all the entries that we don't own have been removed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-9021) Remote query: add option to disable default/legacy indexing per schema file
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-9021?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-9021:
--------------------------------------
Fix Version/s: 9.3.0.CR1
(was: 9.3.0.Beta1)
> Remote query: add option to disable default/legacy indexing per schema file
> ---------------------------------------------------------------------------
>
> Key: ISPN-9021
> URL: https://issues.jboss.org/browse/ISPN-9021
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Adrian Nistor
> Assignee: Adrian Nistor
> Fix For: 9.3.0.CR1, 9.3.0.Final
>
>
> All types that are not annotated are currently fully indexed for backward compat with first version of remote query (which did not have protobuf annotations to control indexing).
> This behaviour is very inefficient and confusing for users that do not intend to use indexing (non-indexed query works anyway).
> Given the history of this behaviour we cannot remove it until next major version. Until then we can offer the choice of disabling it per each schema file via a boolean protobuf file-level option named 'enable_legacy_indexing', which when absent is considered to default to true. Setting it to false will disable indexing of types that do not have indexing annotations.
> Whenever an entry is indexed using default indexing a warning will be logged in order to motivate people to switch to using proper annotations.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months