[JBoss JIRA] (ISPN-8976) 2 subclusters failed to merge to 1 cluster - IllegalLifecycleStateException
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-8976?page=com.atlassian.jira.plugin.... ]
Ryan Emerson updated ISPN-8976:
-------------------------------
Fix Version/s: 9.3.0.Final
Sprint: Sprint 9.3.0.Final
> 2 subclusters failed to merge to 1 cluster - IllegalLifecycleStateException
> ---------------------------------------------------------------------------
>
> Key: ISPN-8976
> URL: https://issues.jboss.org/browse/ISPN-8976
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.1.4.Final
> Reporter: Robert Cernak
> Assignee: Ryan Emerson
> Fix For: 9.3.0.Final
>
> Attachments: logs.zip
>
>
> At the beginning I have main cluster consisted of 8 nodes.
> Then I disconnected main switch on which these nodes were connected.
> This leaded to separating main cluster to 2 subclusters - first with 2 nodes and second with 6 nodes. This was expected.
> After that I rebooted the nodes. After reboot, nodes again correctly formed 2 subclusters with 2 and 6 members.
> After a long time when all nodes were stable with low cpu load, I connected the main switch back which should lead to recreation of main cluster with 8 controllers.
> However main cluster did not recovered:
> subcluster2 did not change - still had 6 nodes connected - no new members
> subcluster1 - nodes did not connect with subcluster2 and after cca 30min they left the cluster.
> When I checked infinispan logs of node1 from 1st subcluster I had IllegalLifecycleStateException for every created cache (see included logs.zip):
> [transport-thread-744a974a-2811-4f79-ac63-f32daf005d7f-p4-t6] (ClusterCacheStatus.java:599) - ISPN000228: Failed to recover cache XXX state after the current node became the coordinator
> org.infinispan.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
[JBoss JIRA] (ISPN-9264) StringBasedStoreInvalidationCacheSyncIT Failures
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-9264?page=com.atlassian.jira.plugin.... ]
Ryan Emerson updated ISPN-9264:
-------------------------------
Description:
{code:java}
testRemoveNonExistentEntry
java.lang.AssertionError: Read timeout at org.infinispan.server.test.cs.jdbc.multinode.StringBasedStoreInvalidationCacheSyncIT.testRemoveNonExistentEntry(StringBasedStoreInvalidationCacheSyncIT.java:82)
{code}
was:
{code}
testRemoveNonExistentEntry
java.lang.AssertionError: Read timeout at org.infinispan.server.test.cs.jdbc.multinode.StringBasedStoreInvalidationCacheSyncIT.testRemoveNonExistentEntry(StringBasedStoreInvalidationCacheSyncIT.java:82)
{code]
> StringBasedStoreInvalidationCacheSyncIT Failures
> ------------------------------------------------
>
> Key: ISPN-9264
> URL: https://issues.jboss.org/browse/ISPN-9264
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.3.0.CR1
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.3.0.Final
>
>
> {code:java}
> testRemoveNonExistentEntry
> java.lang.AssertionError: Read timeout at org.infinispan.server.test.cs.jdbc.multinode.StringBasedStoreInvalidationCacheSyncIT.testRemoveNonExistentEntry(StringBasedStoreInvalidationCacheSyncIT.java:82)
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
[JBoss JIRA] (ISPN-9264) StringBasedStoreInvalidationCacheSyncIT Failures
by Ryan Emerson (JIRA)
Ryan Emerson created ISPN-9264:
----------------------------------
Summary: StringBasedStoreInvalidationCacheSyncIT Failures
Key: ISPN-9264
URL: https://issues.jboss.org/browse/ISPN-9264
Project: Infinispan
Issue Type: Bug
Components: Test Suite - Server
Affects Versions: 9.3.0.CR1
Reporter: Ryan Emerson
Assignee: Ryan Emerson
Fix For: 9.3.0.Final
{code}
testRemoveNonExistentEntry
java.lang.AssertionError: Read timeout at org.infinispan.server.test.cs.jdbc.multinode.StringBasedStoreInvalidationCacheSyncIT.testRemoveNonExistentEntry(StringBasedStoreInvalidationCacheSyncIT.java:82)
{code]
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
[JBoss JIRA] (ISPN-9263) Segmented Off Heap Data Container
by William Burns (JIRA)
William Burns created ISPN-9263:
-----------------------------------
Summary: Segmented Off Heap Data Container
Key: ISPN-9263
URL: https://issues.jboss.org/browse/ISPN-9263
Project: Infinispan
Issue Type: Feature Request
Components: Off Heap
Reporter: William Burns
Assignee: William Burns
Fix For: 9.4.0.Final
We added in an on heap segmented data container with ISPN-5451. We need to also include a segmented off heap data container that implements the new SegmentedDataContainer interface properly.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
[JBoss JIRA] (ISPN-9257) ClustertopologyManagerTest.testAbruptLeaveAfterGetStatus2[SCATTERED_SYNC, tx=false] random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9257?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9257:
-------------------------------
Sprint: Sprint 9.3.0.Final
> ClustertopologyManagerTest.testAbruptLeaveAfterGetStatus2[SCATTERED_SYNC, tx=false] random failures
> ---------------------------------------------------------------------------------------------------
>
> Key: ISPN-9257
> URL: https://issues.jboss.org/browse/ISPN-9257
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.3.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.3.0.Final
>
> Attachments: ISPN-8731_wrong_topology_2018-05-18_ClusterTopologyManagerTest-infinispan-core.log.gz
>
>
> The test kills the coordinator NodeA, then while NodeB is trying to recover the caches it also kills NodeC. It expects NodeB to start a rebalance with 2 nodes and discards it, in order to test that it can process the 1-node rebalance first:
> {noformat}
> 00:34:06,582 DEBUG (transport-thread-test-NodeB-p12-t6:[testCache]) [ClusterTopologyManagerTest] Discarding rebalance command CacheTopology{id=8, phase=TRANSITORY, rebalanceId=5, currentCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (2)[test-NodeB-49590: 85, test-NodeC-58596: 85]}, pendingCH=ScatteredConsistentHash{ns=256, rebalanced=true, owners = (2)[test-NodeB-49590: 128, test-NodeC-58596: 128]}, unionCH=null, actualMembers=[test-NodeB-49590, test-NodeC-58596], persistentUUIDs=[6b96414e-15d8-4350-aa3c-4fb4fc34e888, d47dc4a9-2a95-4bb1-a83b-bb8a27c9999f]}
> 00:34:06,609 DEBUG (transport-thread-test-NodeB-p12-t2:[Topology-testCache]) [LocalTopologyManagerImpl] Updating local topology for cache testCache: CacheTopology{id=9, phase=TRANSITORY, rebalanceId=5, currentCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[test-NodeB-49590: 85]}, pendingCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[test-NodeB-49590: 128]}, unionCH=null, actualMembers=[test-NodeB-49590], persistentUUIDs=[6b96414e-15d8-4350-aa3c-4fb4fc34e888]}
> 00:34:06,609 DEBUG (transport-thread-test-NodeB-p12-t2:[Topology-testCache]) [LocalTopologyManagerImpl] Installing fake cache topology CacheTopology{id=8, phase=NO_REBALANCE, rebalanceId=4, currentCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[test-NodeB-49590: 85]}, pendingCH=null, unionCH=null, actualMembers=[test-NodeB-49590], persistentUUIDs=[6b96414e-15d8-4350-aa3c-4fb4fc34e888]} for cache testCache
> {noformat}
> Unfortunately {{PreferAvailabilityStrategy}} has changed a bit and the rebalance ids don't always match the expectations of the test, so that the 1-node rebalance is discarded instead:
> {noformat}
> 09:46:10,530 DEBUG (transport-thread-Test-NodeB-p54539-t3:[testCache]) [Test] Discarding rebalance command CacheTopology{id=9, phase=TRANSITORY, rebalanceId=5, currentCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[Test-NodeB-62039: 85]}, pendingCH=ScatteredConsistentHash{ns=256, rebalanced=true, owners = (1)[Test-NodeB-62039: 256]}, unionCH=null, actualMembers=[Test-NodeB-62039], persistentUUIDs=[0ed7be74-4485-489b-baee-28c461c9e5de]}
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months