[JBoss JIRA] (ISPN-2240) Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
6:42 p.m.
Robert Stupp created ISPN-2240:
----------------------------------
Summary: Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "] not
held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:42 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp updated ISPN-2240:
-------------------------------
Attachment: somehow.zip
Java files with test case (runs forever)
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:35 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp updated ISPN-2240:
-------------------------------
Attachment: ISPN-2240_fix_TimeoutExceptions.patch
Patch file (5.1.x + master).
I did not verify whether it works in all circumstances - especially when working with
transactions.
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:56 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp updated ISPN-2240:
-------------------------------
Affects Version/s: 5.1.x
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:56 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp commented on ISPN-2240:
------------------------------------
Method
{noformat}
org.infinispan.interceptors.DistributionInterceptor#handleWriteCommand
{noformat}
submits messages unnecessarily to itself via JGroups
There seems to be something going wrong in the distribution code.
Our test case is as follows:
* 4 load generators submit 100 requests at once (4 * 100 parallel threads)
* 4 application servers form an Infinispan cluster
* all app servers are allowed to get and put all cache entries - concurrent updates to the
same cache keys occur
The result is that I see a lot of TimeoutExceptions - all of them look like this:
TimeoutException caused in either an app server thread or a JGroups OOB thread. Either
wants to lock a single key.
Both fail because another local thread is waiting for a distribution response.
Origin locking thread (thread holding the lock):
{noformat}
sun.misc.Unsafe.park Unsafe.java(-2^M
java.util.concurrent.locks.LockSupport.parkNanos LockSupport.java(196^M
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
AbstractQueuedSynchronizer.java(2116^M
org.jgroups.blocks.Request.responsesComplete Request.java(195^M
org.jgroups.blocks.Request.execute Request.java(89^M
org.jgroups.blocks.MessageDispatcher.sendMessage MessageDispatcher.java(366^M
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall
CommandAwareRpcDispatcher.java(275^M
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand
CommandAwareRpcDispatcher.java(165^M
org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely
JGroupsTransport.java(489^M
org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely RpcManagerImpl.java(161^M
org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely RpcManagerImpl.java(183^M
org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely RpcManagerImpl.java(240^M
org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely RpcManagerImpl.java(227^M
org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely RpcManagerImpl.java(222^M
org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely RpcManagerImpl.java(217^M
org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand
DistributionInterceptor.java(512^M
org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand
DistributionInterceptor.java(270^M
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor
PutKeyValueCommand.java(77^M
...
{noformat}
{noformat}
java.lang.Thread.getStackTrace Thread.java(1479^M
org.infinispan.util.concurrent.locks.containers.ReentrantPerEntryLockContainer.tryLock
ReentrantPerEntryLockContainer.java(97^M
org.infinispan.util.concurrent.locks.containers.ReentrantPerEntryLockContainer.tryLock
ReentrantPerEntryLockContainer.java(34^M
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer.acquireLock
AbstractPerEntryLockContainer.java(76^M
org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord
DeadlockDetectingLockManager.java(115^M
org.infinispan.util.concurrent.locks.LockManagerImpl.lock LockManagerImpl.java(209^M
org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck
LockManagerImpl.java(201^M
org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey
AbstractLockingInterceptor.java(114^M
org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand
NonTransactionalLockingInterceptor.java(67^M
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor
PutKeyValueCommand.java(77^M
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor
CommandInterceptor.java(116^M
org.infinispan.interceptors.base.CommandInterceptor.handleDefault
CommandInterceptor.java(130^M
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand
AbstractVisitor.java(62^M
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor
PutKeyValueCommand.java(77^M
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor
CommandInterceptor.java(116^M
...
org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal
InboundInvocationHandlerImpl.java(127^M
org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks
InboundInvocationHandlerImpl.java(136^M
org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry
InboundInvocationHandlerImpl.java(162^M
org.infinispan.remoting.InboundInvocationHandlerImpl.handle
InboundInvocationHandlerImpl.java(114^M
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand
CommandAwareRpcDispatcher.java(226^M
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle
CommandAwareRpcDispatcher.java(203^M
org.jgroups.blocks.RequestCorrelator.handleRequest RequestCorrelator.java(465^M
org.jgroups.blocks.RequestCorrelator.receiveMessage RequestCorrelator.java(372^M
org.jgroups.blocks.RequestCorrelator.receive RequestCorrelator.java(247^M
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up MessageDispatcher.java(601^M
org.jgroups.JChannel.up JChannel.java(715^M
org.jgroups.stack.ProtocolStack.up ProtocolStack.java(1020^M
{noformat}
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:01 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp commented on ISPN-2240:
------------------------------------
Yes - it is definitly at least this case:
# Machine A adds key 1 to the cache
# Machine B adds key 1 to the cache, too
# Machine A sends distribution request
# Machine B sends distribution request
Both calls fail with TimeoutExcepitons
Expected behaviour: both calls go through - (either can "win")
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:08 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp commented on ISPN-2240:
------------------------------------
{noformat}
Machine A Machine B
(master for key X) (master for key Y)
| |
| |
| |
put('X',value) put('X',value)
| |
lock('X') lock('X')
| |
| |
+---->-->-----(JGroups)--------------+
| |
+------------<--<-----(JGroups)---+ |
| |
| |
lock('X') lock('X')
TimeoutException! TimeoutException!
| |
| |
+---------------------------------+ |
| |
+------------------------------------+
| |
| |
throw TimeoutException throw TimeoutException
| |
X X
{noformat}
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:37 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp commented on ISPN-2240:
------------------------------------
The solution could look like this:
# Add a new interceptor which is called before any locking interceptor (call it for
example ModifyingInterceptor)
# Let the ModifyingInterceptor intercept all write commands
# The interceptors' logic shall do:
## If the local node is the primary master, just go through the normal interceptor chain
## If the local node is not the primary master
### Issue the command against the primary master
### Update the local cache according to the result from the primary master (ignore L1
updates from the primary master for this update)
### Return the result from the primary master
This would ensure that no deadlocks due to concurrent updates against the same cache key
can occur because all locking takes place on the primary master.
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:10 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp commented on ISPN-2240:
------------------------------------
Above recommendation works. Would be nice to get some feedback from the developers.
Temporary change of
{noformat}
org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor#visitPutKeyValueCommand
{noformat}
to
{noformat}
public Object visitPutKeyValueCommand(InvocationContext ctx, PutKeyValueCommand
command) throws Throwable {
assertNonTransactional(ctx);
if (dm!=null && rpcManager!=null && ctx.isOriginLocal()) {
Address primary = dm.getPrimaryLocation(command.getKey());
Address local = rpcManager.getAddress();
if (!local.equals(primary)) {
SingleRpcCommand rpcCommand = commandsFactory.buildSingleRpcCommand(command);
Map<Address, Response> response =
rpcManager.invokeRemotely(Collections.singleton(primary), rpcCommand,
ResponseMode.SYNCHRONOUS, 10000);
Response resp = response.get(primary);
if (resp==null) { // todo need to ignore L1 invalidation caused by current
put
entryFactory.wrapEntryForPut(ctx, command.getKey(), null,
!command.isPutIfAbsent());
MVCCEntry e = (MVCCEntry) ctx.lookupEntry(command.getKey());
e.setValue(command.getValue());
e.setLifespan(command.getLifespanMillis());
e.setMaxIdle(command.getMaxIdleTimeMillis());
return null;
}
if (resp instanceof SuccessfulResponse) {
SuccessfulResponse sr = (SuccessfulResponse) resp;
entryFactory.wrapEntryForPut(ctx, command.getKey(), null,
!command.isPutIfAbsent());
MVCCEntry e = (MVCCEntry) ctx.lookupEntry(command.getKey());
e.setValue(command.getValue());
e.setLifespan(command.getLifespanMillis());
e.setMaxIdle(command.getMaxIdleTimeMillis());
return sr.getResponseValue();
}
if (resp instanceof ExceptionResponse) {
ExceptionResponse er = (ExceptionResponse)resp;
throw er.getException();
}
throw new Error("hmmm ... "+resp);
}
}
try {
lockKey(ctx, command.getKey());
return invokeNextInterceptor(ctx, command);
} catch (Throwable te) {
throw cleanLocksAndRethrow(ctx, te);
}
finally {
lockManager.unlockAll(ctx);
}
}
{noformat}
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:14 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robert Stupp commented on ISPN-2240:
------------------------------------
Above code change is not great (L1 cache invalidations come later) - but it shows, that
TimeoutExceptions do no longer occur
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:22 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-2240:
-----------------------------------
Fix Version/s: 5.2.0.Final
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Fix For: 5.2.0.Final
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:02 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Robby Grossman commented on ISPN-2240:
--------------------------------------
Has anybody found a workaround for this in the interim? In the worst case, is there any
timeout I could set so that threads don't hang while waiting for each other? We're
using 5.1.x via Torquebox.
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Fix For: 5.2.0.Final
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:42 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2240:
--------------------------------
Assignee: Manik Surtani (was: Mircea Markus)
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Manik Surtani
Fix For: 5.2.0.Final
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:44 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2240:
-------------------------------------
[~manik] Isn't this the same as ISPN-2381?
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Manik Surtani
Fix For: 5.2.0.Final
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:22 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-2240:
------------------------------------
Mircea, it doesn't seem to be the same as ISPN-2381 to me.
It looks like Robert is using a non-transactional cache, and
NonTransactionalLockingInterceptor still acquires the lock on the originator and on all
the owners - not just on the primary owner.
I'm pretty sure the fix in the description is not correct, because
`l.hasQueuedThreads()` can return false even while another thread is busy working with the
key.
The solution described in
https://issues.jboss.org/browse/ISPN-2240?focusedCommentId=12715426&p...
seems reasonable, mimicking the 2-phase commit in transactional caches. Perhaps doing the
RPCs in the DistributionInterceptor would be more reasonable though.
I'm not sure about the problem with L1 invalidations arriving later, I thought we
explicitly do not send L1 invalidations to the origin of the PutKeyValueCommand, but it
could be that that check is only used when the L1 invalidation threshold is non-zero (0 is
the default). This probably warrants a separate issue.
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Manik Surtani
Fix For: 5.2.0.Final
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:49 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2240:
--------------------------------
Priority: Critical (was: Major)
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Manik Surtani
Priority: Critical
Fix For: 5.2.0.Final
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:08 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2240:
--------------------------------
Fix Version/s: 5.2.0.CR1
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Manik Surtani
Priority: Critical
Fix For: 5.2.0.CR1, 5.2.0.Final
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:08 p.m.
[
https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-2240:
--------------------------------
Assignee: Mircea Markus (was: Manik Surtani)
Per-key lock container leads to superfluous TimeoutExceptions on
concurrent access to same key
----------------------------------------------------------------------------------------------
Key: ISPN-2240
URL: https://issues.jboss.org/browse/ISPN-2240
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Affects Versions: 5.1.6.FINAL, 5.1.x
Reporter: Robert Stupp
Assignee: Mircea Markus
Priority: Critical
Fix For: 5.2.0.CR1, 5.2.0.Final
Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
Hi,
I've encountered a lot of TimeoutExceptions just running a load test against an
infinispan cluster.
I tracked down the reason and found out, that the code in
org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock()
causes these superfluous TimeoutExceptions.
A small test case (which just prints out timeouts, too late timeouts and
"paints" a lot of dots to the console - more dots/second on the console means
better throughput ;-)
In a short test I extended the class ReentrantPerEntryLockContainer and changed the
implementation of releaseLock() as follows:
{noformat}
public void releaseLock(Object lockOwner, Object key) {
ReentrantLock l = locks.get(key);
if (l != null) {
if (!l.isHeldByCurrentThread())
throw new IllegalStateException("Lock for [" + key + "]
not held by current thread " + Thread.currentThread());
while (l.isHeldByCurrentThread())
unlock(l, lockOwner);
if (!l.hasQueuedThreads())
locks.remove(key);
}
else
throw new IllegalStateException("No lock for [" + key +
']');
}
{noformat}
The main improvement is that locks are not removed from the concurrent map as long as
other threads are waiting on that lock.
If the lock is removed from the map while other threads are waiting for it, they may run
into timeouts and force TimeoutExceptions to the client.
The above methods "paints more dots per second" - means: it gives a better
throughput for concurrent accesses to the same key.
The re-implemented method should also fix some replication timeout exceptions.
Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4161
days inactive
4257
days old
infinispan-issues@lists.jboss.org
17 comments
5 participants
participants (5)
-
Dan Berindei (JIRA) -
Galder Zamarreño (JIRA)
-
Mircea Markus (JIRA)
-
Robby Grossman (JIRA)
-
Robert Stupp (JIRA)