[JBoss JIRA] Created: (ISPN-1085) TopologyAwareConsistentHash can run in an endless loop when rehashing
10:47 a.m.
TopologyAwareConsistentHash can run in an endless loop when rehashing
---------------------------------------------------------------------
Key: ISPN-1085
URL: https://issues.jboss.org/browse/ISPN-1085
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 4.2.1.FINAL
Environment: Infinispan 4.2.1 using distributed caches and 3 cluster nodes with
siteId and machineId (siteId is the same on all nodes, machineId is different)
Reporter: Jürgen Kellerer
Assignee: Mircea Markus
I'm not exactly sure how to reproduce it and why it happened but at least the problem
is a quality / error handling issue.
TopologyAwareConsistentHash can run in an endless loop when rehashing runs and
'numOwners' is greater than the available addresses in 'processSequence'.
The endless loop is in the method _getOwners()_ and the reason for it is shown in the
simplified snippet below. It would really be helpful if the method either throws an
exception with an exact definition what went wrong (e.g. including the address list and
numOwners count) or handle the case if it's normal that this can happen when the
cluster is just about to be built up.
{code:java}
private List<Address> getOwners(Address address, int numOwners) {
...
ArrayList<Address> processSequence = ...
...
List<Address> result = new ArrayList<Address>();
while (result.size() < numOwners) {
Iterator<Address> addrIt = processSequence.iterator();
while (addrIt.hasNext()) {
result.add(addrIt.next());
addrIt.remove();
}
}
return result;
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:25 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-1085) TopologyAwareConsistentHash can run in an endless loop when rehashing
[
https://issues.jboss.org/browse/ISPN-1085?page=com.atlassian.jira.plugin....
]
Jürgen Kellerer commented on ISPN-1085:
---------------------------------------
I need to add the _getOwners(...)_ was called through
"getStateProvidersOnLeave()" which does not protect _replCount_ from being
greater than _caches.size();_
Also I discovered that either my setup is somewhat broken or another timing issue occurs
as I have exceptions like "NoSuchElementException" on the attempt to access data
immediately after the node came up caused by the fact that _caches_ and _positions_ is
empty. The exception I got was:
{code}
Caused by: java.util.NoSuchElementException
at java.util.TreeMap.key(TreeMap.java:1223)
at java.util.TreeMap.firstKey(TreeMap.java:284)
at
org.infinispan.distribution.ch.TopologyAwareConsistentHash.getOwner(TopologyAwareConsistentHash.java:141)
at
org.infinispan.distribution.ch.TopologyAwareConsistentHash.locate(TopologyAwareConsistentHash.java:53)
at
org.infinispan.distribution.ch.AbstractConsistentHash.isKeyLocalToAddress(AbstractConsistentHash.java:56)
at
org.infinispan.distribution.DistributionManagerImpl.getLocality(DistributionManagerImpl.java:360)
at
org.infinispan.interceptors.IsMarshallableInterceptor.getMightGoRemote(IsMarshallableInterceptor.java:133)
at
org.infinispan.interceptors.IsMarshallableInterceptor.visitGetKeyValueCommand(IsMarshallableInterceptor.java:70)
at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:64)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76)
at
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:88)
at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:64)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
at org.infinispan.CacheDelegate.get(CacheDelegate.java:209)
{code}
Note: Using "DefaultConsistentHash" this does not happen as the method
_isKeyLocalToAddress_ returns 'false' on empty _positions_ instead of failing.
TopologyAwareConsistentHash can run in an endless loop when
rehashing
---------------------------------------------------------------------
Key: ISPN-1085
URL: https://issues.jboss.org/browse/ISPN-1085
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 4.2.1.FINAL
Environment: Infinispan 4.2.1 using distributed caches and 3 cluster nodes with
siteId and machineId (siteId is the same on all nodes, machineId is different)
Reporter: Jürgen Kellerer
Assignee: Mircea Markus
I'm not exactly sure how to reproduce it and why it happened but at least the problem
is a quality / error handling issue.
TopologyAwareConsistentHash can run in an endless loop when rehashing runs and
'numOwners' is greater than the available addresses in 'processSequence'.
The endless loop is in the method _getOwners()_ and the reason for it is shown in the
simplified snippet below. It would really be helpful if the method either throws an
exception with an exact definition what went wrong (e.g. including the address list and
numOwners count) or handle the case if it's normal that this can happen when the
cluster is just about to be built up.
{code:java}
private List<Address> getOwners(Address address, int numOwners) {
...
ArrayList<Address> processSequence = ...
...
List<Address> result = new ArrayList<Address>();
while (result.size() < numOwners) {
Iterator<Address> addrIt = processSequence.iterator();
while (addrIt.hasNext()) {
result.add(addrIt.next());
addrIt.remove();
}
}
return result;
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:56 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-1085) TopologyAwareConsistentHash can run in an endless loop when rehashing
[
https://issues.jboss.org/browse/ISPN-1085?page=com.atlassian.jira.plugin....
]
Manik Surtani updated ISPN-1085:
--------------------------------
Fix Version/s: 4.2.2.BETA1
5.0.0.CR2
Priority: Critical (was: Major)
TopologyAwareConsistentHash can run in an endless loop when
rehashing
---------------------------------------------------------------------
Key: ISPN-1085
URL: https://issues.jboss.org/browse/ISPN-1085
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 4.2.1.FINAL
Environment: Infinispan 4.2.1 using distributed caches and 3 cluster nodes with
siteId and machineId (siteId is the same on all nodes, machineId is different)
Reporter: Jürgen Kellerer
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.2.BETA1, 5.0.0.CR2
I'm not exactly sure how to reproduce it and why it happened but at least the problem
is a quality / error handling issue.
TopologyAwareConsistentHash can run in an endless loop when rehashing runs and
'numOwners' is greater than the available addresses in 'processSequence'.
The endless loop is in the method _getOwners()_ and the reason for it is shown in the
simplified snippet below. It would really be helpful if the method either throws an
exception with an exact definition what went wrong (e.g. including the address list and
numOwners count) or handle the case if it's normal that this can happen when the
cluster is just about to be built up.
{code:java}
private List<Address> getOwners(Address address, int numOwners) {
...
ArrayList<Address> processSequence = ...
...
List<Address> result = new ArrayList<Address>();
while (result.size() < numOwners) {
Iterator<Address> addrIt = processSequence.iterator();
while (addrIt.hasNext()) {
result.add(addrIt.next());
addrIt.remove();
}
}
return result;
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:59 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-1085) TopologyAwareConsistentHash can run in an endless loop when rehashing
[
https://issues.jboss.org/browse/ISPN-1085?page=com.atlassian.jira.plugin....
]
Manik Surtani updated ISPN-1085:
--------------------------------
Fix Version/s: 5.0.0.CR3
(was: 5.0.0.CR2)
TopologyAwareConsistentHash can run in an endless loop when
rehashing
---------------------------------------------------------------------
Key: ISPN-1085
URL: https://issues.jboss.org/browse/ISPN-1085
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 4.2.1.FINAL
Environment: Infinispan 4.2.1 using distributed caches and 3 cluster nodes with
siteId and machineId (siteId is the same on all nodes, machineId is different)
Reporter: Jürgen Kellerer
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.2.BETA1, 5.0.0.CR3
I'm not exactly sure how to reproduce it and why it happened but at least the problem
is a quality / error handling issue.
TopologyAwareConsistentHash can run in an endless loop when rehashing runs and
'numOwners' is greater than the available addresses in 'processSequence'.
The endless loop is in the method _getOwners()_ and the reason for it is shown in the
simplified snippet below. It would really be helpful if the method either throws an
exception with an exact definition what went wrong (e.g. including the address list and
numOwners count) or handle the case if it's normal that this can happen when the
cluster is just about to be built up.
{code:java}
private List<Address> getOwners(Address address, int numOwners) {
...
ArrayList<Address> processSequence = ...
...
List<Address> result = new ArrayList<Address>();
while (result.size() < numOwners) {
Iterator<Address> addrIt = processSequence.iterator();
while (addrIt.hasNext()) {
result.add(addrIt.next());
addrIt.remove();
}
}
return result;
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:02 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-1085) TopologyAwareConsistentHash can run in an endless loop when rehashing
[
https://issues.jboss.org/browse/ISPN-1085?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-1085:
-------------------------------------
I unsuccessfully tried to reproduce the stack trace. Seems like cache.get is called before
the rehash is finished - which shouldn't be possible. If happens again please take
some context logs to help me reproduce.
TopologyAwareConsistentHash can run in an endless loop when
rehashing
---------------------------------------------------------------------
Key: ISPN-1085
URL: https://issues.jboss.org/browse/ISPN-1085
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 4.2.1.FINAL
Environment: Infinispan 4.2.1 using distributed caches and 3 cluster nodes with
siteId and machineId (siteId is the same on all nodes, machineId is different)
Reporter: Jürgen Kellerer
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.2.BETA1, 5.0.0.CR3
I'm not exactly sure how to reproduce it and why it happened but at least the problem
is a quality / error handling issue.
TopologyAwareConsistentHash can run in an endless loop when rehashing runs and
'numOwners' is greater than the available addresses in 'processSequence'.
The endless loop is in the method _getOwners()_ and the reason for it is shown in the
simplified snippet below. It would really be helpful if the method either throws an
exception with an exact definition what went wrong (e.g. including the address list and
numOwners count) or handle the case if it's normal that this can happen when the
cluster is just about to be built up.
{code:java}
private List<Address> getOwners(Address address, int numOwners) {
...
ArrayList<Address> processSequence = ...
...
List<Address> result = new ArrayList<Address>();
while (result.size() < numOwners) {
Iterator<Address> addrIt = processSequence.iterator();
while (addrIt.hasNext()) {
result.add(addrIt.next());
addrIt.remove();
}
}
return result;
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:19 a.m.
New subject: [JBoss JIRA] Resolved: (ISPN-1085) TopologyAwareConsistentHash can run in an endless loop when rehashing
[
https://issues.jboss.org/browse/ISPN-1085?page=com.atlassian.jira.plugin....
]
Dan Berindei resolved ISPN-1085.
--------------------------------
Fix Version/s: 4.2.2.FINAL
5.0.0.FINAL
Resolution: Done
I merged the pull request. Jurgen, please reopen if you see it happening again.
TopologyAwareConsistentHash can run in an endless loop when
rehashing
---------------------------------------------------------------------
Key: ISPN-1085
URL: https://issues.jboss.org/browse/ISPN-1085
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 4.2.1.FINAL
Environment: Infinispan 4.2.1 using distributed caches and 3 cluster nodes with
siteId and machineId (siteId is the same on all nodes, machineId is different)
Reporter: Jürgen Kellerer
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.2.BETA1, 4.2.2.FINAL, 5.0.0.CR3, 5.0.0.FINAL
I'm not exactly sure how to reproduce it and why it happened but at least the problem
is a quality / error handling issue.
TopologyAwareConsistentHash can run in an endless loop when rehashing runs and
'numOwners' is greater than the available addresses in 'processSequence'.
The endless loop is in the method _getOwners()_ and the reason for it is shown in the
simplified snippet below. It would really be helpful if the method either throws an
exception with an exact definition what went wrong (e.g. including the address list and
numOwners count) or handle the case if it's normal that this can happen when the
cluster is just about to be built up.
{code:java}
private List<Address> getOwners(Address address, int numOwners) {
...
ArrayList<Address> processSequence = ...
...
List<Address> result = new ArrayList<Address>();
while (result.size() < numOwners) {
Iterator<Address> addrIt = processSequence.iterator();
while (addrIt.hasNext()) {
result.add(addrIt.next());
addrIt.remove();
}
}
return result;
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:41 p.m.
New subject: [JBoss JIRA] Commented: (ISPN-1085) TopologyAwareConsistentHash can run in an endless loop when rehashing
[
https://issues.jboss.org/browse/ISPN-1085?page=com.atlassian.jira.plugin....
]
Jürgen Kellerer commented on ISPN-1085:
---------------------------------------
Thanks everyone for fixing this ;-)
I will further investigate why I got the NoSuchElement exception. Currently I also have
trouble in getting the cluster up and running where _all_ nodes can see each other
correctly. If its another Bug I'll come back.
TopologyAwareConsistentHash can run in an endless loop when
rehashing
---------------------------------------------------------------------
Key: ISPN-1085
URL: https://issues.jboss.org/browse/ISPN-1085
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 4.2.1.FINAL
Environment: Infinispan 4.2.1 using distributed caches and 3 cluster nodes with
siteId and machineId (siteId is the same on all nodes, machineId is different)
Reporter: Jürgen Kellerer
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.2.BETA1, 4.2.2.FINAL, 5.0.0.CR3, 5.0.0.FINAL
I'm not exactly sure how to reproduce it and why it happened but at least the problem
is a quality / error handling issue.
TopologyAwareConsistentHash can run in an endless loop when rehashing runs and
'numOwners' is greater than the available addresses in 'processSequence'.
The endless loop is in the method _getOwners()_ and the reason for it is shown in the
simplified snippet below. It would really be helpful if the method either throws an
exception with an exact definition what went wrong (e.g. including the address list and
numOwners count) or handle the case if it's normal that this can happen when the
cluster is just about to be built up.
{code:java}
private List<Address> getOwners(Address address, int numOwners) {
...
ArrayList<Address> processSequence = ...
...
List<Address> result = new ArrayList<Address>();
while (result.size() < numOwners) {
Iterator<Address> addrIt = processSequence.iterator();
while (addrIt.hasNext()) {
result.add(addrIt.next());
addrIt.remove();
}
}
return result;
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
4975
days inactive
4986
days old
infinispan-issues@lists.jboss.org
6 comments
4 participants
participants (4)
-
Dan Berindei (JIRA) -
Jürgen Kellerer (JIRA)
-
Manik Surtani (JIRA)
-
Mircea Markus (JIRA)