1:20 p.m.
Michal Linhard created ISPN-1944:
------------------------------------
Summary: SuspectedException blocks cluster formation
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Manik Surtani
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:22 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration updated ISPN-1944:
------------------------------------------
Bugzilla Update: Perform
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Manik Surtani
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:22 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration updated ISPN-1944:
------------------------------------------
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=806855
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Manik Surtani
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:58 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
Tristan Tarrant commented on ISPN-1944:
---------------------------------------
This is probably one for Dan
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Manik Surtani
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:50 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
Manik Surtani updated ISPN-1944:
--------------------------------
Fix Version/s: 5.1.3.FINAL
Priority: Critical (was: Major)
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=806855 (was:
https://bugzilla.redhat.com/show_bug.cgi?id=806855)
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.3.FINAL
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:50 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
Manik Surtani reassigned ISPN-1944:
-----------------------------------
Assignee: Dan Berindei (was: Manik Surtani)
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Fix For: 5.1.3.FINAL
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:06 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1944:
------------------------------------
With the single target optimization in CommandAwareRpcDispatcher and the changes in
JGroups for JGRP-1428, there are some scenarios now where a SuspectException is thrown
even though we're using the SYNCHRONOUS_IGNORE_LEAVERS ResponseMode.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.3.FINAL
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:30 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
Dan Berindei updated ISPN-1944:
-------------------------------
Attachment: ispn-1944.txt
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.3.FINAL
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:34 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1944:
------------------------------------
After examining the logs closer, it seems the SuspectException is only part of the problem
(and it's actually described in ISPN-1934).
At some point node3 suspects both node01 and node03 and excludes them from the view, but
node01 neither removes node03 from its view nor tries to merge the partitions for 10
minutes (see the attached ispn-1944.txt log).
We need to find whether there is anything that we can change in the JGroups configuration
to prevent this kind of situation. Bela suggested using MERGE3 (see ISPN-1951), but
it's not available in JGroups 3.0.8.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.3.FINAL
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:07 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-1944:
-----------------------------------
Fix Version/s: 5.1.4.CR1
(was: 5.1.3.FINAL)
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:13 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
JBoss JIRA Server <jira-update(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Tristan Tarrant <ttarrant(a)redhat.com> made a comment on jira ISPN-1944
This is probably one for Dan
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:13 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
JBoss JIRA Server <jira-update(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Dan Berindei <dberinde(a)redhat.com> made a comment on jira ISPN-1944
With the single target optimization in CommandAwareRpcDispatcher and the changes in
JGroups for JGRP-1428, there are some scenarios now where a SuspectException is thrown
even though we're using the SYNCHRONOUS_IGNORE_LEAVERS ResponseMode.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:13 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
JBoss JIRA Server <jira-update(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Dan Berindei <dberinde(a)redhat.com> made a comment on jira ISPN-1944
After examining the logs closer, it seems the SuspectException is only part of the problem
(and it's actually described in ISPN-1934).
At some point node3 suspects both node01 and node03 and excludes them from the view, but
node01 neither removes node03 from its view nor tries to merge the partitions for 10
minutes (see the attached ispn-1944.txt log).
We need to find whether there is anything that we can change in the JGroups configuration
to prevent this kind of situation. Bela suggested using MERGE3 (see ISPN-1951), but
it's not available in JGroups 3.0.8.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:35 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Michal Linhard <mlinhard(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
shouldn't version here stay ER4 and only target version be changed to ER7 ? cause I
haven't found this in ER7 yet :-)
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:38 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
mark yarborough <myarboro(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
I think Michal is correct. We accidentally changed "version" instead of
"target release". I will correct to:
version: ER6 or less
target release: ER7
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:17 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
mark yarborough <myarboro(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Technical note added. If any revisions are required, please edit the "Technical
Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services
team.
New Contents:
CCFR - Dan
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:19 a.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
https://issues.jboss.org/browse/ISPN-1934 is fixed, so I think we can close this bug as
well.
We still have the problem with the cluster splitting after a join, but that only means
that the state transfer takes a lot longer than it should. State transfer does eventually
end (even though it doesn't necessarily keep the data consistent, see bu #808623), so
we can't say that a SuspectException blocks the cluster formation any more.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:33 a.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Oops, I thought ISPN-1934 was fixed, but I hadn't issued the pull request yet...
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:07 a.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
Dan Berindei resolved ISPN-1944.
--------------------------------
Resolution: Done
With ISPN-1934 fixed, I'm marking this as resolved as well.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:48 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855] from NEW to ON_DEV
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:48 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
JBoss JIRA Server <jira-update(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Dan Berindei <dberinde(a)redhat.com> updated the status of jira ISPN-1944 to Resolved
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:48 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
JBoss JIRA Server <jira-update(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Dan Berindei <dberinde(a)redhat.com> made a comment on jira ISPN-1944
With ISPN-1934 fixed, I'm marking this as resolved as well.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:28 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855] from ON_DEV to ON_QA
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:33 a.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Michal Linhard <mlinhard(a)redhat.com> changed the Status of [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855] from ON_QA to VERIFIED
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:33 a.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Michal Linhard <mlinhard(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
I haven't seen this in ER7 Testing
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:31 a.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Misha H. Ali <mhusnain(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Technical note updated. If any revisions are required, please edit the "Technical
Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services
team.
Diffed Contents:
@@ -1 +1 @@
-CCFR - Dan+<remark>CCFR - Dan</remark>
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:37 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Michal's log actually shows two problems:
1. When the coordinator commits a cache view, the view should be installed even if some of
the members of the view had already left. This was not happening, but it was fixed in
Infinispan 5.1.4.CR1.
2. The test was just starting a new node. However, the 3rd node somehow got into a
partition by itself and the other nodes didn't notice it even after 10 minutes. This
could be a JGroups problem, but it's very hard to diagnose as this is the only time
we've ever seen it.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:14 a.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
I have tried to replicate the 2nd issue with a JGroups-only test, but I was not able to.
FD_SOCK is enabled in the configuration, so when node C forms a new view by itself it
closes the FD_SOCK connections to A and B, which immediately suspect it and form a new
view by themselves. So I don't think it's possible to have the situation in
thelogs unless there was a real network problem that prevented C from closing the FD_SOCK
connection to A and B gracefully.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:31 a.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
It looks like I was wrong about FD_SOCK, it actually doesn't detect immediately when a
node has left the cluster (and the channel is still up, but seeing only a part of the
cluster).
However, when C leaves the cluster, FD_ALL on C notices that it is the only member and it
stops broadcasting heartbeat messages. After the FD_ALL timeout expires, A and B will
suspect C and eliminate it from their view. So it still shouldn't be possible for A
and B to see view A2 [A, B, C] for 10 minutes after C left...
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:49 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Technical notes:
This bug appeared only once, with version 6.0.0.ER4. A new node joined an existing
cluster, but before state transfer ended it split into a separate network partition and
logged a SuspectException. The existing nodes did not receive a new JGroups view for 10
minutes, after which state transfer failed. The cluster never formed with all 3 members.
There were two separate problems:
* Because of the SuspectException on the joiner, it was never able to form a cluster by
itself. This was fixed in Infinispan 5.1.4.CR1 with
https://issues.jboss.org/browse/ISPN-1934.
* The other two nodes did not install a new JGroups cluster view for 10 minutes, blocking
state transfer for this time. It is very likely that this was fixed in AS7 7.1.3 with
https://issues.jboss.org/browse/AS7-4933.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:14 p.m.
[
https://issues.jboss.org/browse/ISPN-1944?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-1944:
-----------------------------------------------
Misha H. Ali <mhusnain(a)redhat.com> made a comment on [bug
806855|https://bugzilla.redhat.com/show_bug.cgi?id=806855]
Technical note updated. If any revisions are required, please edit the "Technical
Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services
team.
Diffed Contents:
@@ -1 +1,3 @@
-<remark>CCFR - Dan</remark>+When a new node joined an existing JBoss Data
Grid cluster, in some instances the new node splits into a separate network partition
before the state transfer concludes and logged a SuspectException. As a result, the
existing nodes do not receive a new JGroups view for ten minutes, after which the state
transfer fails. The cluster did not form with all three members as expected.
+</para><para>
+This problem occurs rarely, due to two problems. First, the SuspectException on the new
node does not allow a new cluster with just the single new node to form. Secondly, the two
existing nodes in the cluster did not install a new JGroups cluster view for ten minutes,
during which time state transfer remains blocked.
SuspectedException blocks cluster formation
-------------------------------------------
Key: ISPN-1944
URL: https://issues.jboss.org/browse/ISPN-1944
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.2.FINAL
Reporter: Michal Linhard
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.1.4.CR1
Attachments: ispn-1944.txt
This happened in JDG 6.0.0.ER4.
The cluster with first two nodes formed alright and when node3 was joining, it got
SuspectedException for some reason, and then prepare in node01 failed with timeout
(10min).
logs here:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4335
days inactive
4413
days old
infinispan-issues@lists.jboss.org
30 comments
6 participants
participants (6)
-
Dan Berindei (JIRA) -
Galder Zamarreño (JIRA)
-
Manik Surtani (JIRA)
-
Michal Linhard (JIRA)
-
RH Bugzilla Integration (JIRA)
-
Tristan Tarrant (JIRA)