7:28 p.m.
Radim Vansa created ISPN-2580:
---------------------------------
Summary: Do not request segments from all nodes at once
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Mircea Markus
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:30 p.m.
[
https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin....
]
Radim Vansa updated ISPN-2580:
------------------------------
Description:
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
was:
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
Do not request segments from all nodes at once
----------------------------------------------
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Mircea Markus
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:12 p.m.
[
https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2580:
--------------------------------
Fix Version/s: 5.2.0.Beta6
Do not request segments from all nodes at once
----------------------------------------------
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Mircea Markus
Fix For: 5.2.0.Beta6
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:12 p.m.
[
https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2580:
-------------------------------------
How many nodes make a cluster "large"?
does this make the state transfer not to finish? (Bug 881070)
Do not request segments from all nodes at once
----------------------------------------------
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Mircea Markus
Fix For: 5.2.0.Beta6
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:38 p.m.
[
https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin....
]
Radim Vansa commented on ISPN-2580:
-----------------------------------
Yes, the state transfer is not finished in this case. However, I have not marked this as
critical as it is possible that the ST failure may be evaded by another JGroups
configuration, different chunk size etc.
I think I have experienced it with clusters from size 8, but I am not 100% sure, cluster
size 8 might be affected as well (we don't keep track of test runs in Hyperion so I
can't verify that). I am definitely experiencing this in resilience test when I kill
one of 32 nodes and then try to start it and join back.
Yes, 881070 is probably one of the victims as I can see long delays between some of the
StateResponseCommands. But it's hard to confirm without trace logging on org.jgroups.
Do not request segments from all nodes at once
----------------------------------------------
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Mircea Markus
Fix For: 5.2.0.Beta6
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:18 a.m.
[
https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2580:
--------------------------------
Assignee: Adrian Nistor (was: Mircea Markus)
Do not request segments from all nodes at once
----------------------------------------------
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Adrian Nistor
Fix For: 5.2.0.Beta6
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:13 p.m.
[
https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration updated ISPN-2580:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=881070
Do not request segments from all nodes at once
----------------------------------------------
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Adrian Nistor
Fix For: 5.2.0.Beta6
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:13 p.m.
[
https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-2580:
-----------------------------------------------
Radim Vansa <rvansa(a)redhat.com> made a comment on [bug
881070|https://bugzilla.redhat.com/show_bug.cgi?id=881070]
The related issue ISPN-2580 would likely resolve instance of bug reported here.
Do not request segments from all nodes at once
----------------------------------------------
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Adrian Nistor
Fix For: 5.2.0.Beta6
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:44 p.m.
[
https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin....
]
RH Bugzilla Integration commented on ISPN-2580:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug
881070|https://bugzilla.redhat.com/show_bug.cgi?id=881070] from NEW to ASSIGNED
Do not request segments from all nodes at once
----------------------------------------------
Key: ISPN-2580
URL: https://issues.jboss.org/browse/ISPN-2580
Project: Infinispan
Issue Type: Enhancement
Components: State transfer
Affects Versions: 5.2.0.Beta5
Reporter: Radim Vansa
Assignee: Adrian Nistor
Fix For: 5.2.0.Beta6
When a new node joins large cluster filled with data, it gets the new CH and
REBALANCE_START command, and requests data from all nodes at once (or almost all with even
distribution of segments). It may be not able to handle this amount of transfers in
parallel even at the JGroups level - this results in data sent to the node and discarded
at the receiver, sent again and again. With a heavy congestion the node just buffers
fragments of a message from one sender and never passes this up.
The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the
node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4171
days inactive
4172
days old
infinispan-issues@lists.jboss.org
8 comments
3 participants
participants (3)
-
Mircea Markus (JIRA) -
Radim Vansa (JIRA)
-
RH Bugzilla Integration (JIRA)