[JBoss JIRA] (ISPN-4985) Allow JCache TCK to be run exclusively with the main jar
by Galder Zamarreño (JIRA)
Galder Zamarreño created ISPN-4985:
--------------------------------------
Summary: Allow JCache TCK to be run exclusively with the main jar
Key: ISPN-4985
URL: https://issues.jboss.org/browse/ISPN-4985
Project: Infinispan
Issue Type: Bug
Components: JCache
Affects Versions: 7.0.0.Final
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 7.0.1.Final
For TCK certification, we need the main production infinispan-jcache jar to include everything needed to run the TCK. This includes org.infinispan.jcache.test.tck.TckMbeanServerBuilder which currently is only available in the TCK runner project.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months
[JBoss JIRA] (ISPN-4949) Split brain: inconsistent data after merge
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-4949?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on ISPN-4949:
--------------------------------
Here's a possible algorithm similar to the one Dan proposed above that might work for what you want to do in Infinispan:
* View is V5=A,B,C,D
* C and D partition away (or crash)
* A gets view V6=A,B,C
* A creates an AckCollection(A,B,C) for V6 and sends a PREPARE_REBALANCE(V6) to A,B,C
* A receives ACK(V6) from A and B and removes them from the AckCollection
* Since the AckCollection is not empty (C is still in there):
** A doesn't do anything
* When ACK(V6) from C is received: send COMMIT_REBALANCE(V6) to A,B,C
* When a new view V7=A,B is received:
** A sends a PREPARE_REBALANCE(V7) to A,B
** The causes all previous pending PREPARE_REBALANCE to be removed
** Continue as above
* During waiting for all ACKs for a given PREPARE_REBALANCE, the RPC timeouts (if
sync) are reduced, so RPCs return sooner and don't block on crashed members, or
members on the other side of a partition
> Split brain: inconsistent data after merge
> ------------------------------------------
>
> Key: ISPN-4949
> URL: https://issues.jboss.org/browse/ISPN-4949
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Radim Vansa
> Priority: Critical
>
> 1) cluster A, B, C, D splits into 2 parts:
> A, B (coord A) finds this out immediately and enters degraded mode with CH [A, B, C, D]
> C, D (coord D) first detects that B is lost, gets view A, C, D and starts rebalance with CH [A, C, D]. Segment X is primary owned by C (it had backup on B but this got lost)
> 2) D detects that A was lost as well, therefore enters degraded mode with CH [A, C, D]
> 3) C inserts entry into X: all owners (only C) is present, therefore the modification is allowed
> 4) cluster is merged and coordinator finds out that the max stable topology has CH [A, B, C, D] (it is the older of the two partitions' topologies, got from A, B) - logs 'No active or unavailable partitions, so all the partitions must be in degraded mode' (yes, all partitions are in degraded mode, but write has happened in the meantime)
> 5) The old CH is broadcast in newest topology, no rebalance happens
> 6) Inconsistency: read in X may miss the update
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months
[JBoss JIRA] (ISPN-4949) Split brain: inconsistent data after merge
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-4949?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on ISPN-4949 at 11/15/14 2:37 AM:
----------------------------------------------------------
JGroups has had the same view installation algorithm for 15 years now.
In a view A,B,C,D, if both C and D crash or are partitioned away, there's never been a guarantee that A installs view AB immediately, but the view succession can be A,B,C,D -> ABC -> AB *or* A,B,C,D -> AB.
This depends solely on the failure detection protocol used.
We can work with view bundling timeouts, ie. if the crash of C and D are detected *within* the view bundling timeout, then view A,B will likely be installed immediately; if not, there will be a view A,B,C before A,B. But this is not guaranteed.
The fact that Infinispan cannot handle this scenario doesn't mean the view installation algorithm is wrong :-)
I'm not even sure JGRP-1901 makes sense: imagine a partition in a 100 node view: view 1-100 splits into 1-50 and 51-100. If the first partition has to be created going from 1-100 -> 1-50, then the view installation has to be delayed until all members have acked the new view. This means we have to wait until the failure detection suspected members 51-100 before we could install view 1-50 ! This could take a long time, especially if we use FD (over TCP).
During this time, all RPCs from members in the first partition to members in the second partition would block !
This could even get worse: if some members in the second partition are suspended due to GC or temp network problems, then they might come in and out of sight (suspect-unsuspect-suspect), so an algorithm such as outlined in JGRP-1901 would never come to a conclusion and install a new view.
I believe it's better to notify members of a crashed member as soon as possible, so the application can react.
was (Author: belaban):
JGroups has had the same view installation algorithm for 15 years now.
In a view A,B,C,D, if both C and D crash or are partitioned away, there's never been a guarantee that A installs view AB immediately, but the view succession can be A,B,C,D -> ABC -> AB *or* A,B,C,D -> AB.
This depends solely on the failure detection protocol used.
We can work with view bundling timeouts, ie. if the crash of C and D are detected *within* the view bundling timeout, then view A,B will likely be installed immediately; if not, there will be a view A,B,C before A,B. But this is not guaranteed.
The fact that Infinispan cannot handle this scenario doesn't mean the view installation algorithm is wrong :-)
I'm not even sure JGRP-1901 makes sense: imagine a partition in a 100 node view: view 1-100 splits into 1-50 and 51-100. If the first partition has to be created going from 1-100 -> 1-50, then the view installation has to be delayed until all members have acked the new view. This means we have to wait until the failure detection suspected members 51-100 before we could install view 1-50 ! This could take a long time, especially if we use FD (over TCP).
During this time, all RPCs from members in the first partition to members in the second partition would block !
This could even get worse: if some members in the second partition are suspended due to GC or temp network problems, then they might come in and out of sight (suspect-unsuspect-suspect), so an algorithm such as above would never come to a conclusion and install a new view.
I believe it's better to notify members of a crashed member as soon as possible, so the application can react.
> Split brain: inconsistent data after merge
> ------------------------------------------
>
> Key: ISPN-4949
> URL: https://issues.jboss.org/browse/ISPN-4949
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Radim Vansa
> Priority: Critical
>
> 1) cluster A, B, C, D splits into 2 parts:
> A, B (coord A) finds this out immediately and enters degraded mode with CH [A, B, C, D]
> C, D (coord D) first detects that B is lost, gets view A, C, D and starts rebalance with CH [A, C, D]. Segment X is primary owned by C (it had backup on B but this got lost)
> 2) D detects that A was lost as well, therefore enters degraded mode with CH [A, C, D]
> 3) C inserts entry into X: all owners (only C) is present, therefore the modification is allowed
> 4) cluster is merged and coordinator finds out that the max stable topology has CH [A, B, C, D] (it is the older of the two partitions' topologies, got from A, B) - logs 'No active or unavailable partitions, so all the partitions must be in degraded mode' (yes, all partitions are in degraded mode, but write has happened in the meantime)
> 5) The old CH is broadcast in newest topology, no rebalance happens
> 6) Inconsistency: read in X may miss the update
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months
[JBoss JIRA] (ISPN-4949) Split brain: inconsistent data after merge
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-4949?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on ISPN-4949 at 11/15/14 2:37 AM:
----------------------------------------------------------
JGroups has had the same view installation algorithm for 15 years now.
In a view A,B,C,D, if both C and D crash or are partitioned away, there's never been a guarantee that A installs view AB immediately, but the view succession can be A,B,C,D -> ABC -> AB *or* A,B,C,D -> AB.
This depends solely on the failure detection protocol used.
We can work with view bundling timeouts, ie. if the crash of C and D are detected *within* the view bundling timeout, then view A,B will likely be installed immediately; if not, there will be a view A,B,C before A,B. But this is not guaranteed.
The fact that Infinispan cannot handle this scenario doesn't mean the view installation algorithm is wrong :-)
I'm not even sure JGRP-1901 makes sense: imagine a partition in a 100 node view: view 1-100 splits into 1-50 and 51-100. If the first partition has to be created going from 1-100 -> 1-50, then the view installation has to be delayed until all members have acked the new view. This means we have to wait until the failure detection suspected members 51-100 before we could install view 1-50 ! This could take a long time, especially if we use FD (over TCP).
During this time, all RPCs from members in the first partition to members in the second partition would block !
This could even get worse: if some members in the second partition are suspended due to GC or temp network problems, then they might come in and out of sight (suspect-unsuspect-suspect), so an algorithm such as above would never come to a conclusion and install a new view.
I believe it's better to notify members of a crashed member as soon as possible, so the application can react.
was (Author: belaban):
JGroups has had the same view installation algorithm for 15 years now.
In a view A,B,C,D, if both C and D crash or are partitioned away, there's never been a guarantee that A installs view AB immediately, but the view succession can be A,B,C,D -> ABC -> AB *or* A,B,C,D -> AB.
This depends solely on the failure detection protocol used.
We can work with view bundling timeouts, ie. if the crash of C and D are detected *within* the view bundling timeout, then view A,B will likely be installed immediately; if not, there will be a view A,B,C before A,B. But this is not guaranteed.
The fact that Infinispan cannot handle this scenario doesn't mean the view installation algorithm is wrong :-)
I'm not even sure JGRP-1901 makes sense: imagine a partition in a 100 node view: view 1-100 splits into 1-50 and 51-100. If the first partition has to be created going from 1-100 -> 1-50, then the view installation has to be delayed until all members have acked the new view. This means we have to wait until the failure detection suspected members 51-100 before we could install view 1-50 !
During this time, all RPCs from members in the first partition to members in the second partition would block !
This could even get worse: if some members in the second partition are suspended due to GC or temp network problems, then they might come in and out of sight (suspect-unsuspect-suspect), so an algorithm such as above would never come to a conclusion and install a new view.
I believe it's better to notify members of a crashed member as soon as possible, so the application can react.
> Split brain: inconsistent data after merge
> ------------------------------------------
>
> Key: ISPN-4949
> URL: https://issues.jboss.org/browse/ISPN-4949
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Radim Vansa
> Priority: Critical
>
> 1) cluster A, B, C, D splits into 2 parts:
> A, B (coord A) finds this out immediately and enters degraded mode with CH [A, B, C, D]
> C, D (coord D) first detects that B is lost, gets view A, C, D and starts rebalance with CH [A, C, D]. Segment X is primary owned by C (it had backup on B but this got lost)
> 2) D detects that A was lost as well, therefore enters degraded mode with CH [A, C, D]
> 3) C inserts entry into X: all owners (only C) is present, therefore the modification is allowed
> 4) cluster is merged and coordinator finds out that the max stable topology has CH [A, B, C, D] (it is the older of the two partitions' topologies, got from A, B) - logs 'No active or unavailable partitions, so all the partitions must be in degraded mode' (yes, all partitions are in degraded mode, but write has happened in the meantime)
> 5) The old CH is broadcast in newest topology, no rebalance happens
> 6) Inconsistency: read in X may miss the update
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months
[JBoss JIRA] (ISPN-4949) Split brain: inconsistent data after merge
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-4949?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on ISPN-4949:
--------------------------------
JGroups has had the same view installation algorithm for 15 years now.
In a view A,B,C,D, if both C and D crash or are partitioned away, there's never been a guarantee that A installs view AB immediately, but the view succession can be A,B,C,D -> ABC -> AB *or* A,B,C,D -> AB.
This depends solely on the failure detection protocol used.
We can work with view bundling timeouts, ie. if the crash of C and D are detected *within* the view bundling timeout, then view A,B will likely be installed immediately; if not, there will be a view A,B,C before A,B. But this is not guaranteed.
The fact that Infinispan cannot handle this scenario doesn't mean the view installation algorithm is wrong :-)
I'm not even sure JGRP-1901 makes sense: imagine a partition in a 100 node view: view 1-100 splits into 1-50 and 51-100. If the first partition has to be created going from 1-100 -> 1-50, then the view installation has to be delayed until all members have acked the new view. This means we have to wait until the failure detection suspected members 51-100 before we could install view 1-50 !
During this time, all RPCs from members in the first partition to members in the second partition would block !
This could even get worse: if some members in the second partition are suspended due to GC or temp network problems, then they might come in and out of sight (suspect-unsuspect-suspect), so an algorithm such as above would never come to a conclusion and install a new view.
I believe it's better to notify members of a crashed member as soon as possible, so the application can react.
> Split brain: inconsistent data after merge
> ------------------------------------------
>
> Key: ISPN-4949
> URL: https://issues.jboss.org/browse/ISPN-4949
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Radim Vansa
> Priority: Critical
>
> 1) cluster A, B, C, D splits into 2 parts:
> A, B (coord A) finds this out immediately and enters degraded mode with CH [A, B, C, D]
> C, D (coord D) first detects that B is lost, gets view A, C, D and starts rebalance with CH [A, C, D]. Segment X is primary owned by C (it had backup on B but this got lost)
> 2) D detects that A was lost as well, therefore enters degraded mode with CH [A, C, D]
> 3) C inserts entry into X: all owners (only C) is present, therefore the modification is allowed
> 4) cluster is merged and coordinator finds out that the max stable topology has CH [A, B, C, D] (it is the older of the two partitions' topologies, got from A, B) - logs 'No active or unavailable partitions, so all the partitions must be in degraded mode' (yes, all partitions are in degraded mode, but write has happened in the meantime)
> 5) The old CH is broadcast in newest topology, no rebalance happens
> 6) Inconsistency: read in X may miss the update
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months
[JBoss JIRA] (ISPN-4890) Entry iterator should throw AvailabilityException in degraded mode
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4890?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-4890:
-----------------------------------------------
William Burns <wburns(a)redhat.com> changed the Status of [bug 1161783|https://bugzilla.redhat.com/show_bug.cgi?id=1161783] from NEW to POST
> Entry iterator should throw AvailabilityException in degraded mode
> ------------------------------------------------------------------
>
> Key: ISPN-4890
> URL: https://issues.jboss.org/browse/ISPN-4890
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 7.0.0.CR2
> Reporter: Dan Berindei
> Assignee: William Burns
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.0.0.Final
>
>
> When the distributed entry iterator gets a suspect exception, it assumes that the CH will be soon updated to remove the leaver, and retries to contact the primary owner of the segment.
> But if the cache is in degraded mode, the consistent hash is no longer updated, and the entry iterator enters an infinite loop.
> {noformat}
> 11:58:35,622 TRACE (testng-DelayedAvailabilityUpdateTest:) [DistributedEntryRetriever] Request to NodeC-51013=[50, 51, 20, 52, 21, 53, 22, 54, 23, 24, 25, 26, 27, 28, 29] for cd886604-960b-431a-81a0-efbb51eed57f was suspect, must resend to a new node!
> 11:58:35,622 TRACE (testng-DelayedAvailabilityUpdateTest:) [DistributedEntryRetriever] Request to NodeC-51013=[50, 51, 20, 52, 21, 53, 22, 54, 23, 24, 25, 26, 27, 28, 29] for cd886604-960b-431a-81a0-efbb51eed57f was suspect, must resend to a new node!
> 11:58:35,622 TRACE (testng-DelayedAvailabilityUpdateTest:) [DistributedEntryRetriever] Request to NodeC-51013=[50, 51, 20, 52, 21, 53, 22, 54, 23, 24, 25, 26, 27, 28, 29] for cd886604-960b-431a-81a0-efbb51eed57f was suspect, must resend to a new node!
> ...
> {noformat}
> This is visible when running the partition handling tests (e.g. DelayedAvailabilityUpdateTest) with trace logging enabled, as {{TestingUtil.killCaches()}} logs the cache contents at trace level. As an aside, {{TestingUtil.killCaches()}} should probably use {{Flag.CACHE_MODE_LOCAL}} to print only the entries on each node.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months
[JBoss JIRA] (ISPN-4975) Cross site state transfer - status of push gets stuck at "SENDING" after being cancelled
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4975?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-4975:
------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/3078
As suspected, the siteCollector was not cleaned. So, it assumed that the state transfer was always in progress.
It is fixed now and the state transfer is marked as canceled in the status.
> Cross site state transfer - status of push gets stuck at "SENDING" after being cancelled
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-4975
> URL: https://issues.jboss.org/browse/ISPN-4975
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Matej Čimbora
> Assignee: Pedro Ruivo
> Fix For: 7.0.1.Final
>
>
> After invoking: site --cancelpush backupSite on the producer site, status of the push operation seems to get stuck at "SENDING" value (tested by site --pushstatus), even if state transfer is not currently in progress.
> Invoking site --cancelreceive mainSite on the consumer site works correctly. New invocation of site --push backupsite leads to "X-Site state transfer to '%s' already started!" being displayed. The issue seems to be caused by XSiteStateTransferManagerImpl.siteCollector not being cleared.
> Used configuration:
> distributed caches, site A: 2 nodes, site B: 3 nodes, B is a backup for A.
> Scenario
> - Start A,B
> - Take B offline using takeSiteOffline
> - Load data into A
> - Push state into B
> - CancelPushState B
> -- PushStateStatus remains stuck at SENDING & new push is not possible
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months
[JBoss JIRA] (ISPN-4975) Cross site state transfer - status of push gets stuck at "SENDING" after being cancelled
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4975?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-4975:
------------------------------
Fix Version/s: 7.0.1.Final
> Cross site state transfer - status of push gets stuck at "SENDING" after being cancelled
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-4975
> URL: https://issues.jboss.org/browse/ISPN-4975
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Matej Čimbora
> Assignee: Pedro Ruivo
> Fix For: 7.0.1.Final
>
>
> After invoking: site --cancelpush backupSite on the producer site, status of the push operation seems to get stuck at "SENDING" value (tested by site --pushstatus), even if state transfer is not currently in progress.
> Invoking site --cancelreceive mainSite on the consumer site works correctly. New invocation of site --push backupsite leads to "X-Site state transfer to '%s' already started!" being displayed. The issue seems to be caused by XSiteStateTransferManagerImpl.siteCollector not being cleared.
> Used configuration:
> distributed caches, site A: 2 nodes, site B: 3 nodes, B is a backup for A.
> Scenario
> - Start A,B
> - Take B offline using takeSiteOffline
> - Load data into A
> - Push state into B
> - CancelPushState B
> -- PushStateStatus remains stuck at SENDING & new push is not possible
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months
[JBoss JIRA] (ISPN-4975) Cross site state transfer - status of push gets stuck at "SENDING" after being cancelled
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4975?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-4975:
------------------------------
Status: Open (was: New)
> Cross site state transfer - status of push gets stuck at "SENDING" after being cancelled
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-4975
> URL: https://issues.jboss.org/browse/ISPN-4975
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Matej Čimbora
> Assignee: Pedro Ruivo
>
> After invoking: site --cancelpush backupSite on the producer site, status of the push operation seems to get stuck at "SENDING" value (tested by site --pushstatus), even if state transfer is not currently in progress.
> Invoking site --cancelreceive mainSite on the consumer site works correctly. New invocation of site --push backupsite leads to "X-Site state transfer to '%s' already started!" being displayed. The issue seems to be caused by XSiteStateTransferManagerImpl.siteCollector not being cleared.
> Used configuration:
> distributed caches, site A: 2 nodes, site B: 3 nodes, B is a backup for A.
> Scenario
> - Start A,B
> - Take B offline using takeSiteOffline
> - Load data into A
> - Push state into B
> - CancelPushState B
> -- PushStateStatus remains stuck at SENDING & new push is not possible
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 5 months