[JBoss JIRA] (ISPN-2410) A PrepareCommand forwarded back to the originator can time out waiting on a key already locked by itself
3:50 a.m.
Dan Berindei created ISPN-2410:
----------------------------------
Summary: A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command has
to wait for all the backup locks from pending transactions to be released. The problem is
that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:01 a.m.
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-2410:
------------------------------------
TxInterceptor should check if we already have started a prepare for this transaction
(local or remote) and avoid performing the second prepare.
A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
--------------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.Beta3
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:47 a.m.
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-2410:
------------------------------------
A similar problem can happen in non-transactional caches, and without txs we don't
know whether the put command being forwarded back to the originator is the same put
command that is holding the lock locally.
For non-transactional caches, we must not forward a command back to its originator,
because that will lead to a deadlock. However, it's possible that the same command was
already forwarded to us, so there are two nodes we must avoid forwarding the command to.
A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
--------------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.Beta3
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:15 a.m.
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2410:
-------------------------------------
what about the keys for which the tx is now the primary owner? These are backup-locks so
no other tx would be able to lock them? If so I guess that your suggestion with skip
prepare should work.
A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
--------------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.Beta3
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:19 a.m.
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-2410:
------------------------------------
If the originator became the primary owner, it means that either the originator was a
backup owner already (and so had the backup locks) or all the previous owners left the
cache - in which case I don't think we need to worry *too* much about consistency.
The problem I think is with keys for which the originator just became a backup owner. The
commit will assume that the transaction already has those keys in its lookup table, but it
doesn't: the prepare command that was forwarded back created it's own
{{RemoteTransaction}} object, which is isolated from the original {{LocalTransaction}}.
I'm considering renaming {{RemoteTransaction}} to {{CacheTransactionImpl}} and
changing {{LocalTransaction}} to wrap a {{CacheTransaction}} instead of implementing it.
This would allow the local command and the remote one to access the same transaction
object, but it might involve changing a lot of transaction-related code. What do you
think, Mircea?
A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
--------------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.Beta3
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:07 a.m.
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-2410:
------------------------------------
Thinking again about the non-transactional scenario, I'm wondering if we could change
locking so that we don't hold the key lock on the originator while executing the
remote commands. That would eliminate the deadlock.
A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
--------------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.Beta3
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:26 p.m.
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2410:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Beta3)
A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
--------------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:29 a.m.
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2410:
-------------------------------------
[~dan.berindei]the originator already has the prepare information on it. Wouldn't it
be simpler if we explicitly make sure that prepares are never forwarded to originator and
make sure that, when cache topology changes, we iterate on the local transactions and take
care of the lock acquisitions (which is always backup locks as far as I can remember).
This should also solve the problem and reduce the traffic.
A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
--------------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:37 a.m.
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2410:
-------------------------------------
[~dan.berindei] re: renaming, they already inherit from AbstractCacheTransaction, I think
any logic can that is common to both remote and local transactions can be moved there.
Unless you want to implement it as I suggested, then you might not need to do this.
A PrepareCommand forwarded back to the originator can time out
waiting on a key already locked by itself
--------------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:58 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2410:
--------------------------------
Summary: A command forwarded back to the originator can time out waiting on a key
already locked by itself (was: A PrepareCommand forwarded back to the originator can time
out waiting on a key already locked by itself)
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:16 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-2410:
------------------------------------
[~mircea.markus] I'm not sure if you can find an appropriate moment during state
transfer where to do this iteration. But when the prepare command returns to the locking
interceptor, after the RPC is done, we could check if we became an owner for one or more
new keys and acquire those locks locally (or just register backup locks).
I'm creating a separate issue for non-tx caches, but since we don't guarantee
consistency in non-tx caches I think it's ok to avoid the deadlock by forwarding
non-tx write commands asynchronously.
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:48 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2410:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Beta4)
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:45 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2410:
--------------------------------
Priority: Blocker (was: Critical)
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:23 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Dan Berindei reassigned ISPN-2410:
----------------------------------
Assignee: Mircea Markus (was: Dan Berindei)
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:52 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2410:
--------------------------------
Assignee: Mircea Markus (was: Dan Berindei)
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:52 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2410:
--------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:22 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2410:
--------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.2.0.CR1
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:57 a.m.
New subject: [JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
[
https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2410:
--------------------------------
Fix Version/s: 5.2.0.Beta6
(was: 5.2.0.CR1)
A command forwarded back to the originator can time out waiting on a
key already locked by itself
-------------------------------------------------------------------------------------------------
Key: ISPN-2410
URL: https://issues.jboss.org/browse/ISPN-2410
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.2.0.Beta6
If a rebalance happens while a prepare command is executing on a remote node, and the
originator has become an owner, it makes sense to forward the command back to the
originator to lock the keys (or just add them to the backup locks list).
However, we don't keep the old consistent hashes around, so we don't know if the
originator became an owner after invoking the remote command or was already an owner. So
if the topology changed, we always forward the prepare back to the owner.
Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command
has to wait for all the backup locks from pending transactions to be released. The problem
is that we wait for the current transaction as well, causing a deadlock.
Seen in OnePhaseXATest:
{noformat}
18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125
broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value,
flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local,
cacheName='TestCache', topologyId=-1} to recipient list null
18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl]
Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2,
currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl]
Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0,
value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}],
onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote,
cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor]
handleTopologyAffectedCommand for command PrepareCommand
{modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false,
lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false,
gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache',
topologyId=2}, originLocal=false
18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction]
Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key
key0? true
18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication
exception:
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
NodeB-49450
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4399
days inactive
4449
days old
infinispan-issues@lists.jboss.org
17 comments
2 participants
participants (2)
-
Dan Berindei (JIRA) -
Mircea Markus (JIRA)