[jbosscache-issues] [JBoss JIRA] (JBCACHE-1613) Lock TimeoutException when using a JTS transaction manager under load

Lock TimeoutException when using a JTS transaction manager under load --------------------------------------------------------------------- Key: JBCACHE-1613 URL: https://issues.jboss.org/browse/JBCACHE-1613 Project: JBoss Cache Issue Type: Bug Security Level: Public(Everyone can see) Components: Transactions Affects Versions: 3.2.8.GA Reporter: Dennis Reed Assignee: Dennis Reed Fix For: 3.2.9.GA Attachments: JTSCache.java After switching to JTS for transactions, "random" lock timeouts occur: org.jboss.cache.lock.TimeoutException: Unable to acquire lock on Fqn [/foo] after [15000] milliseconds for requestor [GlobalTransaction:<10.1.2.3:12345>:5]! Lock held by [null] at org.jboss.cache.mvcc.MVCCNodeHelper.acquireLock(MVCCNodeHelper.java:157) The actual cause is only logged at TRACE level: TRACE [org.jboss.cache.lock.MVCCLockManager] (RequestProcessor-97) Attempting to unlock /foo TRACE [org.jboss.cache.lock.MVCCLockManager] (RequestProcessor-97) Caught exception and ignoring. java.lang.IllegalMonitorStateException: org.jboss.cache.util.concurrent.locks.OwnableReentrantLock@4d93ddfa[State = 1, empty queue][Locked by GlobalTransaction:<10.1.2.3:12345>:4] at org.jboss.cache.util.concurrent.locks.OwnableReentrantLock.tryRelease(OwnableReentrantLock.java:139) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239) at org.jboss.cache.util.concurrent.locks.OwnableReentrantLock.unlock(OwnableReentrantLock.java:105) at org.jboss.cache.util.concurrent.locks.PerElementLockContainer.releaseLock(PerElementLockContainer.java:112) at org.jboss.cache.lock.MVCCLockManager.unlock(MVCCLockManager.java:147) at org.jboss.cache.interceptors.MVCCLockingInterceptor.cleanupLocks(MVCCLockingInterceptor.java:375) at org.jboss.cache.interceptors.MVCCLockingInterceptor.transactionalCleanup(MVCCLockingInterceptor.java:400) at org.jboss.cache.interceptors.MVCCLockingInterceptor.handlePrepareCommand(MVCCLockingInterceptor.java:332) ... The root cause is that JTS commits transactions in a different thread than the one calling commit. Under load, it even uses different threads for each call during the same transaction commit (beforeCompletion/afterCompletion). JBoss Cache stores some data in ThreadLocals, and when different threads call beforeCompletion and afterCompletion for the same transaction, it causes it to mix the data between the transactions causing data corruption. Particularly the "ctx = invocationContextContainer.get();" calls in org.jboss.cache.interceptors.TxInterceptor beforeCompletion and afterCompletion.