[JBoss JIRA] (ISPN-2383) Transactions incorrectly removed by the StaleTransactionCleanupService causing inconsistencies
8:32 p.m.
Mircea Markus created ISPN-2383:
-----------------------------------
Summary: Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Critical
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST but
only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:10 a.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2383:
--------------------------------
Assignee: Adrian Nistor (was: Mircea Markus)
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:05 a.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2383:
--------------------------------
Assignee: Mircea Markus (was: Adrian Nistor)
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Critical
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:56 p.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Adrian Nistor commented on ISPN-2383:
-------------------------------------
To clarify, you propose to not start StaleTransactionCleanupService on topology change
event as it is implemented now because it is too early but rather wait until all transfers
are completed and only invoke it then.
Also please note we might need to pause StaleTransactionCleanupService when state transfer
starts again.
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Critical
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:26 p.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Adrian Nistor edited comment on ISPN-2383 at 10/12/12 1:24 PM:
---------------------------------------------------------------
To clarify, you propose to not start StaleTransactionCleanupService on topology change
event as it is implemented now because it is too early but rather wait until all transfers
are completed and only invoke it then.
Also please note we might need to pause StaleTransactionCleanupService when state transfer
starts again, so listening for topology change events is still needed.
was (Author: anistor):
To clarify, you propose to not start StaleTransactionCleanupService on topology change
event as it is implemented now because it is too early but rather wait until all transfers
are completed and only invoke it then.
Also please note we might need to pause StaleTransactionCleanupService when state transfer
starts again.
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Critical
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:50 p.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2383:
-------------------------------------
@ [~anistor] yes, that's what I had in mind.
re: pausing the StaleTransactionCleanupService, is that needed in order to handle
overlapping state transfers?
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Critical
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:08 a.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2383:
--------------------------------
Priority: Blocker (was: Critical)
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:08 a.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2383:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1388
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:50 p.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2383:
-------------------------------------
the solution implemented was to keep track of the nodes where the transaction prepares and
send the rollback/commit to both these nodes and the new owners. Also the
StaleTransactionCleanupService is amended to only remoce transactions if the owners are no
longer in the cluster, rest of transactions being removed explicitly by the
rollback/commit commands.
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:27 p.m.
[
https://issues.jboss.org/browse/ISPN-2383?page=com.atlassian.jira.plugin....
]
Dan Berindei updated ISPN-2383:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
Transactions incorrectly removed by the
StaleTransactionCleanupService causing inconsistencies
-----------------------------------------------------------------------------------------------
Key: ISPN-2383
URL: https://issues.jboss.org/browse/ISPN-2383
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Mircea Markus
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
- tx runs on N1, prepare a single-key tx (k1) on N2
- block commit execution on N2 and start N3 so that k1 node maps to N3
- the StaleTransactionCleanupService[1] removes the transaction from N2
- when commit is executed there's no tx on N2 to forward the commit to so it simply
does nothing
- the TxCompletionNotification arrives on N3 and rollbacks the uncommitted data
- data is lost!
Solution: the StaleTransactionCleanupService should not run concurrently with the NBST
but only after the NBST is finished. I.e. it should no longer be an topology listener, but
should be invoked explicitly during the NBST-complete RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4484
days inactive
4487
days old
infinispan-issues@lists.jboss.org
9 comments
3 participants
participants (3)
-
Adrian Nistor (JIRA) -
Dan Berindei (JIRA)
-
Mircea Markus (JIRA)