2:55 p.m.
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Fix For: 4.2.0.CR1
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce:
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:59 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Vladimir Blagojevic updated ISPN-777:
-------------------------------------
Attachment: CacheScheduledCounter.java
ISPN-777_output.txt
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce:
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:01 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Vladimir Blagojevic updated ISPN-777:
-------------------------------------
Description:
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
was:
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce:
1) Kill one running instance.
2) Restart it
See attached example output of a run.
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:06 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani updated ISPN-777:
-------------------------------
Priority: Blocker (was: Major)
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:44 a.m.
New subject: [JBoss JIRA] Work started: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Work on ISPN-777 started by Mircea Markus.
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:52 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus commented on ISPN-777:
------------------------------------
Here is what is causing this. A and B two nodes.
- A start a remote lock acquisition on B and then fails
- there is a thread (T1) on B that tries to acquire a remote lock on behalf of A
- on A's leave, there is another thread that starts on B (T2) that tries to cleanup
all the unfinished transactions started by A
the problem is that T2 runs before T1. This way the lock is acquire and never released.
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:58 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus commented on ISPN-777:
------------------------------------
As a solution I'm thinking to either: keep track of all the remote threads that run on
a node. The lock cleanup thread in TxTable would first interrupt all these threads and
only then run the tx cleanup command. This relies on the fact that an interrupted thread
would not be able to acquire locks anymore(needs to be enforced through a unit test).
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:58 a.m.
New subject: [JBoss JIRA] Issue Comment Edited: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus edited comment on ISPN-777 at 11/22/10 6:58 AM:
--------------------------------------------------------------
As a solution: keep track of all the remote threads that run on a node. The lock cleanup
thread in TxTable would first interrupt all these threads and only then run the tx cleanup
command. This relies on the fact that an interrupted thread would not be able to acquire
locks anymore(needs to be enforced through a unit test).
was (Author: mircea.markus):
As a solution I'm thinking to either: keep track of all the remote threads that
run on a node. The lock cleanup thread in TxTable would first interrupt all these threads
and only then run the tx cleanup command. This relies on the fact that an interrupted
thread would not be able to acquire locks anymore(needs to be enforced through a unit
test).
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:15 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus commented on ISPN-777:
------------------------------------
Bela suggested another solution at jgroups level:
#1 Keep track of the threads going into user space
#2 on up(VIEW): interrupt all of the threads in the list and wait for their return (time
bounded)
This is similar to BARRIER (take a look)
13:09 mmarkus bela: yep, I think that should work. Do you need this in jgroups code base?
13:10 bela not necessarily somewhere in Infinispan would probably be better
This way, you can release this in lock-step with the ISPN release schedule
I would suggest you insert the protocol dynamically at config time, based on yoru config
e,g, if you don't run TXs, you won't need the protocol
13:11 mmarkus 13:11 bela: +1. I'll add this as an solution and have thought about it
vs having the login in ISPN directly
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:24 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus commented on ISPN-777:
------------------------------------
Another solution from bela:
bela
Before a TX acquires locks, it should check if its owner is still in the current view
If not, the TX would be released
This way, even of T2 runs after T1, T1 would commit suicide
13:17
mmarkus
bela: yes. but this would be an check-then-act that should be guarded with a lock
13:17
bela
and roll itself back - because it is not in the current view instaled by T1
OK, so T1 and T2 access the TX table
You guarantee serialized access through a lock
So it's either T1 --> T2, or T2 --> T1
T2 first cleans up running TXs and the view is already set
T1 then sees that it is not in the current view and rolls back itself
or
T1 sees that it's in the current view and proceeds
T2 the cleans it up
13:20
mmarkus
that would work as well.
13:20
bela
I'd give it a simple try, seems easier than the prev solution
13:21
mmarkus
So i would need to have a single thread working on behalf of a transaction at a time
bela: which I think it makes sense
bela: +1.
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:20 a.m.
New subject: [JBoss JIRA] Resolved: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus resolved ISPN-777.
--------------------------------
Resolution: Done
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:27 a.m.
New subject: [JBoss JIRA] Reopened: (ISPN-777) Eager cluster wide locks not cleaned upon rollback
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani reopened ISPN-777:
--------------------------------
Assignee: Manik Surtani (was: Mircea Markus)
Eager cluster wide locks not cleaned upon rollback
--------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:19 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani updated ISPN-777:
-------------------------------
Summary: Race conditions in cleaning up stale locks held by dead members in a
cluster (was: Eager cluster wide locks not cleaned upon rollback)
Complexity: High
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:19 a.m.
New subject: [JBoss JIRA] Work started: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Work on ISPN-777 started by Manik Surtani.
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:21 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani updated ISPN-777:
-------------------------------
JBoss Forum Reference: http://community.jboss.org/thread/158844?tstart=0
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:39 a.m.
New subject: [JBoss JIRA] Resolved: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani resolved ISPN-777.
--------------------------------
Resolution: Done
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:01 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani commented on ISPN-777:
------------------------------------
See http://lists.jboss.org/pipermail/infinispan-dev/2010-December/006943.html for details
of the fix
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:40 a.m.
New subject: [JBoss JIRA] Reopened: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani reopened ISPN-777:
--------------------------------
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:22 p.m.
New subject: [JBoss JIRA] Commented: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://jira.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.sys...
]
Galder Zamarreño commented on ISPN-777:
---------------------------------------
Oh oh, we have a problem here. Not sure Manik why you reopened it, but while doing some
work to verify ISPN-244, I've spotted that RemoteLockCleanupStressTest throws. Note
that this is not related to the ISPN-244 cos the issue is present in 4.2.x as well where
ISPN-244 has no bearing:
2010-12-08 18:16:25,945 13951 ERROR
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(Incoming-2,Infinispan-Cluster,NodeC-2057:) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: java.io.EOFException: Read past end
of file
at
org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:331)
at
org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:102)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:598)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:712)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:772)
at org.jgroups.JChannel.up(JChannel.java:1422)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:954)
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:478)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:525)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:464)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:225)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:190)
at org.jgroups.protocols.FC.up(FC.java:483)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:888)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:615)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:295)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:707)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
at org.jgroups.protocols.FD.up(FD.java:266)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:269)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:210)
at org.jgroups.protocols.Discovery.up(Discovery.java:292)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1093)
at org.jgroups.protocols.TP.access$100(TP.java:56)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1633)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1615)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.EOFException: Read past end of file
at org.jboss.marshalling.AbstractUnmarshaller.eofOnRead(AbstractUnmarshaller.java:184)
at
org.jboss.marshalling.AbstractUnmarshaller.readUnsignedByteDirect(AbstractUnmarshaller.java:319)
at
org.jboss.marshalling.AbstractUnmarshaller.readUnsignedByte(AbstractUnmarshaller.java:280)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
at
org.infinispan.marshall.jboss.GenericJBossMarshaller.objectFromObjectStream(GenericJBossMarshaller.java:162)
at
org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:184)
at
org.infinispan.statetransfer.StateTransferManagerImpl.processCommitLog(StateTransferManagerImpl.java:228)
at
org.infinispan.statetransfer.StateTransferManagerImpl.applyTransactionLog(StateTransferManagerImpl.java:250)
at
org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:320)
... 29 more
Now, in spite of this issue the test passes!!! (WTF?).
So, first things first, the test needs improving to send these issues back up. That
probably means changing to a Callable rather than Runnable, and will see which exceptions
we wanna swallow and which ones we wanna propagate. Then I'll get going to figure out
the cause. I'm creating a JIRA to deal with this.
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://jira.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR1
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:14 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://issues.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.s...
]
Manik Surtani updated ISPN-777:
-------------------------------
Fix Version/s: 4.2.0.CR4
(was: 4.2.0.CR1)
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://issues.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR4
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:30 a.m.
New subject: [JBoss JIRA] Resolved: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://issues.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.s...
]
Manik Surtani resolved ISPN-777.
--------------------------------
Resolution: Done
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://issues.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.0.CR4
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:11 a.m.
New subject: [JBoss JIRA] Reopened: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://issues.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.s...
]
Manik Surtani reopened ISPN-777:
--------------------------------
Reopening as this needs further investigation as per the user forum thread
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://issues.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.1.Final
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:11 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://issues.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.s...
]
Manik Surtani updated ISPN-777:
-------------------------------
Fix Version/s: 4.2.1.Final
(was: 4.2.0.CR4)
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://issues.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Blocker
Fix For: 4.2.1.Final
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:25 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-777) Race conditions in cleaning up stale locks held by dead members in a cluster
[
https://issues.jboss.org/browse/ISPN-777?page=com.atlassian.jira.plugin.s...
]
Manik Surtani updated ISPN-777:
-------------------------------
Fix Version/s: 4.2.0.Final
(was: 4.2.1.Final)
Priority: Critical (was: Blocker)
Correcting this. This issue was fixed in 4.2.0.FINAL.
Race conditions in cleaning up stale locks held by dead members in a
cluster
----------------------------------------------------------------------------
Key: ISPN-777
URL: https://issues.jboss.org/browse/ISPN-777
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 4.2.0.BETA1
Reporter: Vladimir Blagojevic
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.2.0.Final
Attachments: CacheScheduledCounter.java, ISPN-777_output.txt
It seems that rollback sometimes does not release acquired eager locks. See attached test
program and run two JVM instances on the same machine. Program schedules a task to run
every 5 seconds. Tasks simply locks a key, gets the value, increases the value and puts it
back surrounded by begin/commit/rollback tx boundary.
Steps to reproduce (keep repeating steps until problem is encountered):
1) Kill one running instance.
2) Restart it
See attached example output of a run.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4845
days inactive
4907
days old
infinispan-issues@lists.jboss.org
23 comments
4 participants
participants (4)
-
Galder Zamarreño (JIRA) -
Manik Surtani (JIRA)
-
Mircea Markus (JIRA)
-
Vladimir Blagojevic (JIRA)