[JBoss JIRA] (ISPN-2483) State transfer issue with the transactions for which the originator has crashed
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2483?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2483:
--------------------------------
Assignee: Dan Berindei (was: Adrian Nistor)
> State transfer issue with the transactions for which the originator has crashed
> -------------------------------------------------------------------------------
>
> Key: ISPN-2483
> URL: https://issues.jboss.org/browse/ISPN-2483
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Transactions
> Affects Versions: 5.1.8.Final, 5.2.0.Beta3
> Reporter: Mircea Markus
> Assignee: Dan Berindei
> Priority: Blocker
> Fix For: 5.2.0.Beta6, 5.2.0.Final
>
>
> State transfer migrates and prepares the transactions for which the originator has left. On the receiving node, this results in the transaction being prepared and acquiring backup locks which are never released (unless manual intervention).
> This should behave as follows:
> - if there's no recovery enabled, the state producer should not send such transactions but drop them
> - if recovery is enabled these transactions should be sent across. They shouldn't be prepared/acquire backup locks, but be placed in the recovery cache (see RecoveryManagerImpl.inDoubtTransactions)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
[JBoss JIRA] (ISPN-2580) Do not request segments from all nodes at once
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2580:
--------------------------------
Assignee: Dan Berindei (was: Adrian Nistor)
> Do not request segments from all nodes at once
> ----------------------------------------------
>
> Key: ISPN-2580
> URL: https://issues.jboss.org/browse/ISPN-2580
> Project: Infinispan
> Issue Type: Enhancement
> Components: State transfer
> Affects Versions: 5.2.0.Beta5
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 5.2.0.Beta6
>
>
> When a new node joins large cluster filled with data, it gets the new CH and REBALANCE_START command, and requests data from all nodes at once (or almost all with even distribution of segments). It may be not able to handle this amount of transfers in parallel even at the JGroups level - this results in data sent to the node and discarded at the receiver, sent again and again. With a heavy congestion the node just buffers fragments of a message from one sender and never passes this up.
> The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
[JBoss JIRA] (ISPN-2550) NoSuchElementException in Hot Rod Encoder
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2550?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-2550:
----------------------------------------
Tomas, seems like the config that you provided works fine as storage only.
Can you create a separate issue to follow your issue? Don't wanna mix with node kill issue.
Also, if you can replicate the issue again and provide JDG version information, TRACE logs...etc? Can you try to replicate the issue on master of JDG too?
> NoSuchElementException in Hot Rod Encoder
> -----------------------------------------
>
> Key: ISPN-2550
> URL: https://issues.jboss.org/browse/ISPN-2550
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta4
> Reporter: Michal Linhard
> Assignee: Galder Zamarreño
> Priority: Blocker
> Fix For: 5.2.0.Beta6
>
>
> Tomas noticed this a while ago in a specific functional test:
> https://bugzilla.redhat.com/show_bug.cgi?id=875151
> I'm creating a more general JIRA, cause I'm having this in resilience test.
> What I found by quick debug, is that here:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
> {code}
> for (segmentIdx <- 0 until numSegments) {
> val denormalizedSegmentHashIds = allDenormalizedHashIds(segmentIdx)
> val segmentOwners = ch.locateOwnersForSegment(segmentIdx)
> for (ownerIdx <- 0 until segmentOwners.length) {
> val address = segmentOwners(ownerIdx % segmentOwners.size)
> val serverAddress = members(address)
> val hashId = denormalizedSegmentHashIds(ownerIdx)
> log.tracef("Writing hash id %d for %s:%s", hashId, serverAddress.host, serverAddress.port)
> writeString(serverAddress.host, buf)
> writeUnsignedShort(serverAddress.port, buf)
> buf.writeInt(hashId)
> }
> }
> {code}
> we're trying to obtain serverAddress for nonexistent address and NoSuchElementException is not handled properly.
> It hapens after I kill a node in a resilience test and the exception appears when querying for the node in the members cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
[JBoss JIRA] (ISPN-2550) NoSuchElementException in Hot Rod Encoder
by Michal Linhard (JIRA)
[ https://issues.jboss.org/browse/ISPN-2550?page=com.atlassian.jira.plugin.... ]
Michal Linhard commented on ISPN-2550:
--------------------------------------
The IndexOutOfBoundsException was found when running with https://github.com/danberindei/infinispan/commit/c3325b134704016fa5563435...
btw now i can see another commit on the t_2550_m branch, would it still be helpful to test with it ?
> NoSuchElementException in Hot Rod Encoder
> -----------------------------------------
>
> Key: ISPN-2550
> URL: https://issues.jboss.org/browse/ISPN-2550
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta4
> Reporter: Michal Linhard
> Assignee: Galder Zamarreño
> Priority: Blocker
> Fix For: 5.2.0.Beta6
>
>
> Tomas noticed this a while ago in a specific functional test:
> https://bugzilla.redhat.com/show_bug.cgi?id=875151
> I'm creating a more general JIRA, cause I'm having this in resilience test.
> What I found by quick debug, is that here:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
> {code}
> for (segmentIdx <- 0 until numSegments) {
> val denormalizedSegmentHashIds = allDenormalizedHashIds(segmentIdx)
> val segmentOwners = ch.locateOwnersForSegment(segmentIdx)
> for (ownerIdx <- 0 until segmentOwners.length) {
> val address = segmentOwners(ownerIdx % segmentOwners.size)
> val serverAddress = members(address)
> val hashId = denormalizedSegmentHashIds(ownerIdx)
> log.tracef("Writing hash id %d for %s:%s", hashId, serverAddress.host, serverAddress.port)
> writeString(serverAddress.host, buf)
> writeUnsignedShort(serverAddress.port, buf)
> buf.writeInt(hashId)
> }
> }
> {code}
> we're trying to obtain serverAddress for nonexistent address and NoSuchElementException is not handled properly.
> It hapens after I kill a node in a resilience test and the exception appears when querying for the node in the members cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
[JBoss JIRA] (ISPN-2546) StateChunk with isLastChunk=true not sent when all entries are sent ahead
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2546?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2546:
-----------------------------------------------
Radim Vansa <rvansa(a)redhat.com> changed the Status of [bug 879570|https://bugzilla.redhat.com/show_bug.cgi?id=879570] from ON_QA to VERIFIED
> StateChunk with isLastChunk=true not sent when all entries are sent ahead
> -------------------------------------------------------------------------
>
> Key: ISPN-2546
> URL: https://issues.jboss.org/browse/ISPN-2546
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.Beta4
> Reporter: Radim Vansa
> Assignee: Adrian Nistor
> Priority: Blocker
> Fix For: 5.2.0.Beta5, 5.2.0.Final
>
>
> During a state transfer the entries are sent batched to chunks. However, if all entries are sent in the non-last chunk, the list in entriesBySegment.get(segmentId) is empty. The code for sending last chunks is following:
> {code:title=OutboundTransferTask.sendEntries(...)}
> ...
> if (isLast) {
> for (int segmentId : segments) {
> List<InternalCacheEntry> entries = entriesBySegment.get(segmentId);
> if (entries == null) {
> chunks.add(new StateChunk(segmentId, InfinispanCollections.<InternalCacheEntry>emptyList(), true));
> }
> }
> }
> ...
> {code}
> See that the check is {{entries == null}} but not {{entries.isEmpty()}}.
> This causes to leave some segments unfinished, never finishing the state transfer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
[JBoss JIRA] (ISPN-2550) NoSuchElementException in Hot Rod Encoder
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2550?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2550:
------------------------------------
Michal, what is the last commit you had when you ran the test?
> NoSuchElementException in Hot Rod Encoder
> -----------------------------------------
>
> Key: ISPN-2550
> URL: https://issues.jboss.org/browse/ISPN-2550
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta4
> Reporter: Michal Linhard
> Assignee: Galder Zamarreño
> Priority: Blocker
> Fix For: 5.2.0.Beta6
>
>
> Tomas noticed this a while ago in a specific functional test:
> https://bugzilla.redhat.com/show_bug.cgi?id=875151
> I'm creating a more general JIRA, cause I'm having this in resilience test.
> What I found by quick debug, is that here:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
> {code}
> for (segmentIdx <- 0 until numSegments) {
> val denormalizedSegmentHashIds = allDenormalizedHashIds(segmentIdx)
> val segmentOwners = ch.locateOwnersForSegment(segmentIdx)
> for (ownerIdx <- 0 until segmentOwners.length) {
> val address = segmentOwners(ownerIdx % segmentOwners.size)
> val serverAddress = members(address)
> val hashId = denormalizedSegmentHashIds(ownerIdx)
> log.tracef("Writing hash id %d for %s:%s", hashId, serverAddress.host, serverAddress.port)
> writeString(serverAddress.host, buf)
> writeUnsignedShort(serverAddress.port, buf)
> buf.writeInt(hashId)
> }
> }
> {code}
> we're trying to obtain serverAddress for nonexistent address and NoSuchElementException is not handled properly.
> It hapens after I kill a node in a resilience test and the exception appears when querying for the node in the members cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month
[JBoss JIRA] (ISPN-2566) TopologyAwareConsistentHashFactory rebalance doesn't redistribute data properly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2566?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2566:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 868832|https://bugzilla.redhat.com/show_bug.cgi?id=868832] from ON_DEV to MODIFIED
> TopologyAwareConsistentHashFactory rebalance doesn't redistribute data properly
> -------------------------------------------------------------------------------
>
> Key: ISPN-2566
> URL: https://issues.jboss.org/browse/ISPN-2566
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.0.Beta4
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 5.2.0.Beta6, 5.2.0.Final
>
>
> Say we have a topology-aware cache with numOwners = 2 and two nodes: A(machine=m1) and B(machine=m1). When node C(machine=m2) joins, it should own every key, either as a primary or as a backup owner. This doesn't happen, node C owns just as many segments as nodes A and B.
> Example:
> {noformat}
> 19:21:17,295 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (undefined) Updating cache topology topology for rebalance:
> CacheTopology{id=3, currentCH=DefaultConsistentHash{numSegments=80, numOwners=2,
> members=[node0/default(primary), node1/default(primary)],
> owners={0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1,
> 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0 1, 13: 0 1, 14: 0 1, 15: 0 1,
> 16: 0 1, 17: 0 1, 18: 0 1, 19: 0 1, 20: 0 1, 21: 0 1, 22: 0 1, 23: 0 1,
> 24: 0 1, 25: 0 1, 26: 0 1, 27: 0 1, 28: 0 1, 29: 0 1, 30: 0 1, 31: 0 1,
> 32: 0 1, 33: 0 1, 34: 0 1, 35: 0 1, 36: 0 1, 37: 0 1, 38: 0 1, 39: 0 1,
> 40: 1 0, 41: 1 0, 42: 1 0, 43: 1 0, 44: 1 0, 45: 1 0, 46: 1 0, 47: 1 0,
> 48: 1 0, 49: 1 0, 50: 1 0, 51: 1 0, 52: 1 0, 53: 1 0, 54: 1 0, 55: 1 0,
> 56: 1 0, 57: 1 0, 58: 1 0, 59: 1 0, 60: 1 0, 61: 1 0, 62: 1 0, 63: 1 0,
> 64: 1 0, 65: 1 0, 66: 1 0, 67: 1 0, 68: 1 0, 69: 1 0, 70: 1 0, 71: 1 0,
> 72: 1 0, 73: 1 0, 74: 1 0, 75: 1 0, 76: 1 0, 77: 1 0, 78: 1 0, 79: 1 0},
> pendingCH=DefaultConsistentHash{numSegments=80, numOwners=2,
> members=[node0/default(primary), node1/default(primary), node2/default(secondary)],
> owners={0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1,
> 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0 1, 13: 0 1, 14: 0 1, 15: 0 1,
> 16: 0 1, 17: 0 1, 18: 0 1, 19: 0 1, 20: 0 1, 21: 0 1, 22: 0 1, 23: 0 1,
> 24: 0 1, 25: 0 1, 26: 0 1, 27: 2 0, 28: 2 0, 29: 2 0, 30: 2 0, 31: 2 0,
> 32: 2 0, 33: 2 0, 34: 2 0, 35: 2 0, 36: 2 0, 37: 2 0, 38: 2 0, 39: 2 0,
> 40: 1 0, 41: 1 0, 42: 1 0, 43: 1 0, 44: 1 0, 45: 1 0, 46: 1 0, 47: 1 0,
> 48: 1 0, 49: 1 0, 50: 1 0, 51: 1 0, 52: 1 0, 53: 1 0, 54: 1 0, 55: 1 0,
> 56: 1 0, 57: 1 0, 58: 1 0, 59: 1 0, 60: 1 0, 61: 1 0, 62: 1 0, 63: 1 0,
> 64: 1 0, 65: 1 0, 66: 1 0, 67: 2 1, 68: 2 1, 69: 2 1, 70: 2 1, 71: 2 1,
> 72: 2 1, 73: 2 1, 74: 2 1, 75: 2 1, 76: 2 1, 77: 2 1, 78: 2 1, 79: 2 1}}
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 1 month