12:43 p.m.
Erik Salter created ISPN-2357:
---------------------------------
Summary: New coordinator installs old CH containing crashed members
Key: ISPN-2357
URL: https://issues.jboss.org/browse/ISPN-2357
Project: Infinispan
Issue Type: Feature Request
Components: State transfer
Affects Versions: 5.2.0.Alpha4
Reporter: Erik Salter
Assignee: Mircea Markus
After merging the round of pull requests today (including ISPN-2349), there still is an
issue where a killed node can appear in the CH even after JGroups passes up the new view.
This appears to happen when the old coordinator was killed.
Obviously, if the CH contains the old member, writes that map to the segments previously
owned by the dead node will fail.
Here are ISPN trace logs. You are looking for erm-cluster.
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg1)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.84/server.l... (phl-dg2)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg3,
the node that was killed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:31 a.m.
[
https://issues.jboss.org/browse/ISPN-2357?page=com.atlassian.jira.plugin....
]
Erik Salter commented on ISPN-2357:
-----------------------------------
After looking at the trace logs, it looks like the coordinator dying abruptly fails to
send a LEAVE ClusterControlCommand, so the first thing the new coordinator does is do a
GET_STATUS without consulting the view it received.
Maybe the first thing the new coordinator should do is ask the nodes to reconcile their
current views with the views received from JGroupsTransport?
New coordinator installs old CH containing crashed members
----------------------------------------------------------
Key: ISPN-2357
URL: https://issues.jboss.org/browse/ISPN-2357
Project: Infinispan
Issue Type: Feature Request
Components: State transfer
Affects Versions: 5.2.0.Alpha4
Reporter: Erik Salter
Assignee: Mircea Markus
After merging the round of pull requests today (including ISPN-2349), there still is an
issue where a killed node can appear in the CH even after JGroups passes up the new view.
This appears to happen when the old coordinator was killed.
Obviously, if the CH contains the old member, writes that map to the segments previously
owned by the dead node will fail.
Here are ISPN trace logs. You are looking for erm-cluster.
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg1)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.84/server.l... (phl-dg2)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg3,
the node that was killed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:56 p.m.
[
https://issues.jboss.org/browse/ISPN-2357?page=com.atlassian.jira.plugin....
]
Dan Berindei reassigned ISPN-2357:
----------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
New coordinator installs old CH containing crashed members
----------------------------------------------------------
Key: ISPN-2357
URL: https://issues.jboss.org/browse/ISPN-2357
Project: Infinispan
Issue Type: Feature Request
Components: State transfer
Affects Versions: 5.2.0.Alpha4
Reporter: Erik Salter
Assignee: Dan Berindei
After merging the round of pull requests today (including ISPN-2349), there still is an
issue where a killed node can appear in the CH even after JGroups passes up the new view.
This appears to happen when the old coordinator was killed.
Obviously, if the CH contains the old member, writes that map to the segments previously
owned by the dead node will fail.
Here are ISPN trace logs. You are looking for erm-cluster.
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg1)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.84/server.l... (phl-dg2)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg3,
the node that was killed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:04 p.m.
[
https://issues.jboss.org/browse/ISPN-2357?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-2357:
------------------------------------
This should be handled on the coordinator - it should use its own JGroups view to prune
leavers from the consistent hashes. After all, it couldn't be the coordinator if it
still had the old coordinator in its JGroups view.
Sorry I missed this, it looks like our test suite always gives the coordinator enough time
to install the new topology before shutting down.
New coordinator installs old CH containing crashed members
----------------------------------------------------------
Key: ISPN-2357
URL: https://issues.jboss.org/browse/ISPN-2357
Project: Infinispan
Issue Type: Feature Request
Components: State transfer
Affects Versions: 5.2.0.Alpha4
Reporter: Erik Salter
Assignee: Dan Berindei
After merging the round of pull requests today (including ISPN-2349), there still is an
issue where a killed node can appear in the CH even after JGroups passes up the new view.
This appears to happen when the old coordinator was killed.
Obviously, if the CH contains the old member, writes that map to the segments previously
owned by the dead node will fail.
Here are ISPN trace logs. You are looking for erm-cluster.
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg1)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.84/server.l... (phl-dg2)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg3,
the node that was killed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:22 p.m.
[
https://issues.jboss.org/browse/ISPN-2357?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2357:
--------------------------------
Fix Version/s: 5.2.0.Beta1
New coordinator installs old CH containing crashed members
----------------------------------------------------------
Key: ISPN-2357
URL: https://issues.jboss.org/browse/ISPN-2357
Project: Infinispan
Issue Type: Feature Request
Components: State transfer
Affects Versions: 5.2.0.Alpha4
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.2.0.Beta1
After merging the round of pull requests today (including ISPN-2349), there still is an
issue where a killed node can appear in the CH even after JGroups passes up the new view.
This appears to happen when the old coordinator was killed.
Obviously, if the CH contains the old member, writes that map to the segments previously
owned by the dead node will fail.
Here are ISPN trace logs. You are looking for erm-cluster.
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg1)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.84/server.l... (phl-dg2)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg3,
the node that was killed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:59 p.m.
[
https://issues.jboss.org/browse/ISPN-2357?page=com.atlassian.jira.plugin....
]
Dan Berindei updated ISPN-2357:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1358
When a node becomes the coordinator, it should not include nodes that
that didn't respond to the cache status request in the consistent hash.
Synchronize broadcasting of CH_UPDATE commands so that it's not possible
for the cluster members to receive them out-of-order.
New coordinator installs old CH containing crashed members
----------------------------------------------------------
Key: ISPN-2357
URL: https://issues.jboss.org/browse/ISPN-2357
Project: Infinispan
Issue Type: Feature Request
Components: State transfer
Affects Versions: 5.2.0.Alpha4
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.2.0.Beta1
After merging the round of pull requests today (including ISPN-2349), there still is an
issue where a killed node can appear in the CH even after JGroups passes up the new view.
This appears to happen when the old coordinator was killed.
Obviously, if the CH contains the old member, writes that map to the segments previously
owned by the dead node will fail.
Here are ISPN trace logs. You are looking for erm-cluster.
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg1)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.84/server.l... (phl-dg2)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg3,
the node that was killed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:27 a.m.
[
https://issues.jboss.org/browse/ISPN-2357?page=com.atlassian.jira.plugin....
]
Adrian Nistor updated ISPN-2357:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
Integrated.
New coordinator installs old CH containing crashed members
----------------------------------------------------------
Key: ISPN-2357
URL: https://issues.jboss.org/browse/ISPN-2357
Project: Infinispan
Issue Type: Feature Request
Components: State transfer
Affects Versions: 5.2.0.Alpha4
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.2.0.Beta1
After merging the round of pull requests today (including ISPN-2349), there still is an
issue where a killed node can appear in the CH even after JGroups passes up the new view.
This appears to happen when the old coordinator was killed.
Obviously, if the CH contains the old member, writes that map to the segments previously
owned by the dead node will fail.
Here are ISPN trace logs. You are looking for erm-cluster.
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg1)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.84/server.l... (phl-dg2)
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA4/relay2/10.30.12.83/server.l... (phl-dg3,
the node that was killed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4234
days inactive
4238
days old
infinispan-issues@lists.jboss.org
6 comments
4 participants
participants (4)
-
Adrian Nistor (JIRA) -
Dan Berindei (JIRA)
-
Erik Salter (JIRA)
-
Mircea Markus (JIRA)