7:21 a.m.
Radim Vansa created ISPN-2316:
---------------------------------
Summary: Distributed deadlock in StateTransferInterceptor
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Feature Request
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is somehow delayed (for one minute) in the interceptor
chain between InvocationContextInterceptor and OptimisticLockingInterceptor (it looks like
the only place is waiting for StateTransferInterceptor transactionLock (shared), however,
I cannot find any trace that it is held by anyone)
After the transaction commit times out on old node releasing the lock, the GET_TRANSACTION
request may continue, but the state transfer itself can also timeout if not set properly
longer.
Note that the transaction commit continues on the new node after the ST times out, until
it is found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:21 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Radim Vansa updated ISPN-2316:
------------------------------
Description:
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is somehow delayed (for one minute - until ST times out)
in the interceptor chain between InvocationContextInterceptor and
OptimisticLockingInterceptor (it looks like the only place is waiting for
StateTransferInterceptor transactionLock (shared), however, I cannot find any trace that
it is held by anyone)
After the transaction commit times out on old node releasing the lock, the GET_TRANSACTION
request may continue, but the state transfer itself can also timeout if not set properly
longer.
Note that the transaction commit continues on the new node after the ST times out, until
it is found invalid (rolled back).
was:
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is somehow delayed (for one minute) in the interceptor
chain between InvocationContextInterceptor and OptimisticLockingInterceptor (it looks like
the only place is waiting for StateTransferInterceptor transactionLock (shared), however,
I cannot find any trace that it is held by anyone)
After the transaction commit times out on old node releasing the lock, the GET_TRANSACTION
request may continue, but the state transfer itself can also timeout if not set properly
longer.
Note that the transaction commit continues on the new node after the ST times out, until
it is found invalid (rolled back).
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Feature Request
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is somehow delayed (for one minute - until ST times
out) in the interceptor chain between InvocationContextInterceptor and
OptimisticLockingInterceptor (it looks like the only place is waiting for
StateTransferInterceptor transactionLock (shared), however, I cannot find any trace that
it is held by anyone)
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
Note that the transaction commit continues on the new node after the ST times out, until
it is found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:08 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Radim Vansa updated ISPN-2316:
------------------------------
Description:
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in shared
way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the GET_TRANSACTION
request may continue, but the state transfer itself can also timeout if not set properly
longer.
The transaction commit continues on the new node after the ST times out, until it is found
invalid (rolled back).
was:
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is somehow delayed (for one minute - until ST times out)
in the interceptor chain between InvocationContextInterceptor and
OptimisticLockingInterceptor (it looks like the only place is waiting for
StateTransferInterceptor transactionLock (shared), however, I cannot find any trace that
it is held by anyone)
After the transaction commit times out on old node releasing the lock, the GET_TRANSACTION
request may continue, but the state transfer itself can also timeout if not set properly
longer.
Note that the transaction commit continues on the new node after the ST times out, until
it is found invalid (rolled back).
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Feature Request
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:10 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Radim Vansa commented on ISPN-2316:
-----------------------------------
I have updated the description as I have finally found the lock (adding thread writing all
thread's stacks periodically to trace can be pretty useful).
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Feature Request
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:22 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Radim Vansa updated ISPN-2316:
------------------------------
Issue Type: Bug (was: Feature Request)
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:24 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Radim Vansa updated ISPN-2316:
------------------------------
Priority: Critical (was: Major)
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
Priority: Critical
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:34 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Radim Vansa commented on ISPN-2316:
-----------------------------------
Happened for example in
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/ispn-52-radargun-res...
if this is of your interest (however, the participants in the deadlock are slave3 as the
requester for GET_TRANSACTIONS and slave2 as the committer).
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
Priority: Critical
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
9 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Radim Vansa commented on ISPN-2316:
-----------------------------------
In the hudson job above there is another interesting fact: As the slave3 requests
GET_TRANSACTIONS in thread OOB-23 at 11:12:50,997, this request is processed on slave2 by
OOB-52 at 11:12:51,002 and delayed until 11:13:51,003 when the StressorThread-7 times out
performing the rpc back to slave3 and releases its locks (then the transactions are
retrieved as indicated by 11:13:51,003 TRACE
[org.infinispan.statetransfer.StateProviderImpl] (OOB-52,edg-perf02-60212) Found 20
transaction(s) to transfer). However, the thread OOB-23 on slave3 does not receive the
response and it stays locked (see stack traces for this thread after 11:13:51 when it
should already receive the response).
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
Priority: Critical
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:08 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Radim Vansa commented on ISPN-2316:
-----------------------------------
Another weird situation, this time in
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/ispn-52-radargun-res...
:
* at 08:07:48,271 thread OOB-11 on slave4 requests transactions from slave2
* this is carried at 08:07:48,275 by thread OOB-45 on slave2, the thread is waiting for
transactionLock
* nevertheless, the OOB-11 on slave4 gets response at 08:08:09,313, in fact failing to get
the transactions. StateTransferTimeout is set to 10 minutes, so having the connection
timed out is not possible
* shortly after that at 08:08:09,713 the slave4 gets new cluster view with itself only
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Mircea Markus
Priority: Critical
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:35 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-2316:
-----------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.Beta1
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:35 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-2316:
-----------------------------------
Fix Version/s: 5.2.0.Beta1
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.Beta1
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:43 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2316:
--------------------------------
Priority: Blocker (was: Critical)
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.2.0.Beta1
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:04 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Dan Berindei updated ISPN-2316:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1342
Reorganize the StateTransferLock:
Remove the transactions lock and the commands lock.
Split waitForTopology() into waitForTopology() and waitForTransactionData()
Introduce a new read-write topology lock to replace the commands lock.
It's ok to remove the transactions lock because any transaction
modification (after the transactions table snapshot was taken)
is guaranteed to be forwarded to the new owners.
waitForTopology() is now used only to ensure that when joiners start requesting
state, every other node already knows about them.
waitForTransactionData() is used to block commands until the local node
has received transaction data for all the new segments.
The new read-write topology lock is held exclusively during topology changes
and non-exclusively during commits to the data container - so that data not
owned by the local node is not written to the data container.
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.2.0.Beta1
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:57 a.m.
[
https://issues.jboss.org/browse/ISPN-2316?page=com.atlassian.jira.plugin....
]
Adrian Nistor updated ISPN-2316:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
Integrated.
Distributed deadlock in StateTransferInterceptor
------------------------------------------------
Key: ISPN-2316
URL: https://issues.jboss.org/browse/ISPN-2316
Project: Infinispan
Issue Type: Bug
Components: State transfer, Transactions
Affects Versions: 5.2.0.Alpha3
Reporter: Radim Vansa
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.2.0.Beta1
When using transactions, a distributed deadlock may occur when a node is joining under
these circumstances:
1) the new node requests transactions using GET_TRANSACTIONS
2) the old node tries to commit a transaction, broadcasting PrepareCommand - in
StateTransferIntreceptor it locks the transactionLock in shared way
3) the request GET_TRANSACTIONS comes on the new node, the node is waiting for the
transactionLock (it requires it exclusively)
4) transaction commit on new node is waiting for the commandsLock (requires this in
shared way) but it is locked exclusively by the onTopologyUpdate - addTransfer -
requestTransactions ( = synchronous GET_TRANSACTIONS).
Found in some traces, but not required:
After the transaction commit times out on old node releasing the lock, the
GET_TRANSACTION request may continue, but the state transfer itself can also timeout if
not set properly longer.
The transaction commit continues on the new node after the ST times out, until it is
found invalid (rolled back).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4469
days inactive
4479
days old
infinispan-issues@lists.jboss.org
13 comments
5 participants
participants (5)
-
Adrian Nistor (JIRA) -
Dan Berindei (JIRA)
-
Galder Zamarreño (JIRA)
-
Mircea Markus (JIRA)
-
Radim Vansa (JIRA)