noon
Erik Salter created ISPN-2291:
---------------------------------
Summary: Tx rollback during state transfer has stale locks
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Mircea Markus
There are stale locks that happened when a transaction failed due to a replication timeout
to its peer node. The transaction contained grouped keys that were submitted to the
primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:04 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Erik Salter commented on ISPN-2291:
-----------------------------------
I used the following to check for stale keys:
for k in "$1" ;
do zcat lock/*/server.log*.gz | grep -F $k | egrep "Unable to acquire|Successfully
acquired lock|Attempting to unlock" > $1 ; done
cat $1 | sed -n 's/\(.*\) \(DEBUG\|TRACE\) .*\(Fail\|acquired\|unlock\).*/\1 \3/p'
| less
A sample key is ServiceGroupKey[edgeDeviceId=4,serviceGroupNo=435]. (All these
*[edgeDeviceId=*] keys use edgeDeviceId as their "group").
You can find the described flow above on lock/10.30.12.85/server.gz on
(OOB-7,erm-cluster,phl-dg3-41250)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Mircea Markus
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:23 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-2291:
-----------------------------------
Component/s: State transfer
(was: Core API)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Mircea Markus
Fix For: 5.2.0.Alpha4
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:23 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-2291:
-----------------------------------
Fix Version/s: 5.2.0.Alpha4
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Mircea Markus
Fix For: 5.2.0.Alpha4
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:23 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-2291:
-----------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.2.0.Alpha4
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:45 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Erik Salter commented on ISPN-2291:
-----------------------------------
More trace logs from different runs. (Quickly encountered, so the logs are small)
Run 1:
https://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.lo...
https://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.lo...
https://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.lo...
Run 2:
https://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/SNAPSHOT/10.30.12.83/serve...
https://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/SNAPSHOT/10.30.12.84/serve...
https://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/SNAPSHOT/10.30.12.85/serve...
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.2.0.Alpha4
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:36 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2291:
--------------------------------
Fix Version/s: 5.2.0.Beta1
(was: 5.2.0.Alpha4)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.2.0.Beta1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:05 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Dan Berindei reassigned ISPN-2291:
----------------------------------
Assignee: Adrian Nistor (was: Dan Berindei)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Adrian Nistor
Fix For: 5.2.0.Beta1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:06 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2291:
--------------------------------
Priority: Critical (was: Major)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.Beta1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:31 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2291:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Beta1)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:56 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Erik Salter commented on ISPN-2291:
-----------------------------------
Here are logs from 5.2.0.Beta1 (or Alpha5), after the recent round of NBST fixes. In this
test, I killed the coordinator (10.30.12.84) The stale lock in this case is:
ServiceGroupKey[edgeDeviceId=2,serviceGroupNo=201]
http://dl.dropbox.com/u/50401510/5.2.0.BETA1/stalelock/10.30.12.83/server...
http://dl.dropbox.com/u/50401510/5.2.0.BETA1/stalelock/10.30.12.84/server...
http://dl.dropbox.com/u/50401510/5.2.0.BETA1/stalelock/10.30.12.85/server...
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:09 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Erik Salter updated ISPN-2291:
------------------------------
Affects Version/s: 5.2.0.Beta1
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3, 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:43 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Erik Salter commented on ISPN-2291:
-----------------------------------
Part of the issue is caused by the xsite.BaseBackupInterceptor and the fact that it
inherits BaseRpcManager, which has its own LockControlCommand impl. This causes the lock
command to be broadcasted to ALL members, not a subset. So on a rollback (and maybe even
commit), the command is only sent to a subset.
This can be mitigated by moving the base LockControlCommand handler to the derived
InvaidationInterceptors and ReplicationInterceptors.
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3, 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:17 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Erik Salter commented on ISPN-2291:
-----------------------------------
So here's the root cause -- the StaleTransactionCleanupService can remove transactions
from the tx table while there are pending locks. I've seen 3 cases of this during a
rehash:
1. A RemoteTransaction was registered and the lock was attempting to be acquired. The tx
is cleaned up by the StaleTxCleanupService in another thread since there were no locks yet
acquired. Afterwards, the invocation context acquires the lock and registers it anyway.
2. Similar to the above, but the invocation context is waiting in the interceptor on the
pending locks (lockKeyAndCheckOwnership). By the time it finishes waiting on the pending
locks, the tx has been cleaned up.
3. I've seen the tx get cleaned up AS SOON as the RemoteTransaction is registered
with the txTable.
..
2012-10-06 18:15:14,249 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl]
(OOB-19,erm-cluster,phl-dg3-8896(phl)) Calling perform() on
LockControlCommand{cache=serviceGroup,
keys=[ServiceGroupKey[edgeDeviceId=4,serviceGroupNo=401]], flags=null, unlock=false}
2012-10-06 18:15:14,249 TRACE [org.infinispan.transaction.TransactionTable]
(OOB-19,erm-cluster,phl-dg3-8896(phl)) Created and registered remote transaction
RemoteTransaction{modifications=[], lookedUpEntries={}, lockedKeys= null, backupKeyLocks
null, missingLookedUpEntries false,
tx=GlobalTransaction:<phl-dg2-51640(phl)>:8900:remote}
...
2012-10-06 18:15:14,250 TRACE [org.infinispan.transaction.StaleTransactionCleanupService]
(transport-thread-20) Killing remote transaction without any local keys
GlobalTransaction:<phl-dg2-51640(phl)>:8900:remote
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3, 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:59 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2291:
-------------------------------------
Integrated https://github.com/infinispan/infinispan/pull/1365 but not closing the JIRA as
it doesn't solves all the problems.
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3, 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:50 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-2291:
-----------------------------------
Affects Version/s: (was: 5.2.0.Alpha3)
Forum Reference:
http://lists.jboss.org/pipermail/infinispan-dev/2012-October/011402.html
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:52 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2291:
-------------------------------------
So my understanding of the problem is: a belated lock acquisition command (be it lock
acquisition or prepare) acquires some locks on keys that no longer map to this node. These
locks are never released. Is that correct?
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:10 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2291:
--------------------------------
Priority: Blocker (was: Critical)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Blocker
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:37 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2291:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Beta2)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Adrian Nistor
Priority: Blocker
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:21 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2291:
--------------------------------
Assignee: Mircea Markus (was: Adrian Nistor)
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:13 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-2291:
------------------------------------
Erik, I think the issues you describe in your last comment are fixed by ISPN-2383 in the
case where the local node is no longer the owner (because StaleTransactionCleanupService
doesn't roll back the transaction any more, instead we wait for the commit/rollback
from the originator).
On the other hand, the StaleTransactionCleanupService still rolls back the tx if the
originator has left the cluster. If the local node was running a lock or prepare command
for that tx (e.g. because the execution thread was waiting for a lock to be released by
another tx), that command would eventually acquire the lock and assign it to a stale
transaction. So we still need this part of your patch.
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:41 p.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Adrian Nistor commented on ISPN-2291:
-------------------------------------
We have an unresolved issue ISPN-711 regarding transactions w/o modifications not
releasing locks - completely unrelated to NBST but possibly related to the issue observed
by you. Does your transaction on ServiceGroupKey[edgeDeviceId=X,serviceGroupNo=Y] make any
changes or it just obtains an explicit lock?
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:42 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2291:
-------------------------------------
[~anistor] there was still a race condition that ISPN-2383 didn't catch.
I've issued a pull request with it.
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:42 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2291:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1396
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:44 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Mircea Markus edited comment on ISPN-2291 at 10/16/12 2:43 AM:
---------------------------------------------------------------
[~anistor] there was still a race condition that ISPN-2383 didn't catch.
I've issued a pull request with it. Indeed it's very much similar with how Erik
fixed it. Just that Erik's fix, whilst releasing the locks, doesn't clean up the
tx from the table as well.
was (Author: mircea.markus):
[~anistor] there was still a race condition that ISPN-2383 didn't catch.
I've issued a pull request with it.
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.CR1
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:52 a.m.
[
https://issues.jboss.org/browse/ISPN-2291?page=com.atlassian.jira.plugin....
]
Dan Berindei updated ISPN-2291:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
Integrated in master.
Tx rollback during state transfer has stale locks
-------------------------------------------------
Key: ISPN-2291
URL: https://issues.jboss.org/browse/ISPN-2291
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta1
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Blocker
Fix For: 5.2.0.Beta3
There are stale locks that happened when a transaction failed due to a replication
timeout to its peer node. The transaction contained grouped keys that were submitted to
the primary owner. During execution of one such transaction, there was a state transfer,
which changed ownership of these keys.
Here's an example flow:
The task executed and started a transaction. Since it was not the primary owner, it sent
a LockControlCommand to the new owner. This call timed out, and a rollback was issued.
The local transaction is completed, but subsequent attempts to lock this key fail.
The logs can be found here:
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.83/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.84/server.log.gz
http://dl.dropbox.com/u/50401510/5.2.0.ALPHA3/lock/10.30.12.85/server.log.gz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4444
days inactive
4486
days old
infinispan-issues@lists.jboss.org
25 comments
5 participants
participants (5)
-
Adrian Nistor (JIRA) -
Dan Berindei (JIRA)
-
Erik Salter (JIRA)
-
Galder Zamarreño (JIRA)
-
Mircea Markus (JIRA)