[jbosscache-issues] [JBoss JIRA] Resolved: (JBCACHE-1497) When adding node, CacheLoaderInterceptor locks child node before MVCCLockingInterceptor locks parent

When adding node, CacheLoaderInterceptor locks child node before MVCCLockingInterceptor locks parent ---------------------------------------------------------------------------------------------------- Key: JBCACHE-1497 URL: https://jira.jboss.org/jira/browse/JBCACHE-1497 Project: JBoss Cache Issue Type: Bug Security Level: Public(Everyone can see) Components: Cache loaders, Locking Affects Versions: 3.0.3.GA Reporter: Krzysztof Sobolewski Assignee: Manik Surtani Fix For: 3.1.0.GA Attachments: AddRemoveNodeDeadlockTest.java I finally got to the root cause of those deadlocks I mentioned in this forum thread: http://www.jboss.org/index.html?module=bb&op=viewtopic&t=152123 It really is this issue of inverted locking order, but somewhere else than I originally looked. Turns out this only happen with cache loader configured: it's the CacheLoaderInterceptor that locks the child before locking parent, as is evident in these two stack traces: The child was locked first with this stack: java.lang.Exception: TRACE PREV at org.jboss.cache.lock.MVCCLockManager.lockAndRecord(MVCCLockManager.java:132) at org.jboss.cache.mvcc.MVCCNodeHelper.acquireLock(MVCCNodeHelper.java:155) at org.jboss.cache.mvcc.MVCCNodeHelper.wrapNodeForWriting(MVCCNodeHelper.java:241) at org.jboss.cache.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:300) at org.jboss.cache.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:165) at org.jboss.cache.interceptors.ActivationInterceptor.visitGetKeyValueCommand(ActivationInterceptor.java:132) at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131) at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110) at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301) at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283) at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110) at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:77) at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178) at org.jboss.cache.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:130) at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110) at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97) at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287) at org.jboss.cache.invocation.CacheInvocationDelegate.get(CacheInvocationDelegate.java:444) Then, the parent: ava.lang.Exception: TRACE at org.jboss.cache.lock.MVCCLockManager.lockAndRecord(MVCCLockManager.java:132) at org.jboss.cache.mvcc.MVCCNodeHelper.acquireLock(MVCCNodeHelper.java:155) at org.jboss.cache.mvcc.MVCCNodeHelper.wrapNodeForWriting(MVCCNodeHelper.java:223) at org.jboss.cache.interceptors.MVCCLockingInterceptor.handlePutKeyValueCommand(MVCCLockingInterceptor.java:101) at org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.visitPutKeyValueCommand(PrePostProcessingCommandInterceptor.java:88) at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:137) at org.jboss.cache.interceptors.ActivationInterceptor.visitPutKeyValueCommand(ActivationInterceptor.java:146) at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131) at org.jboss.cache.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:65) at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301) at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283) at org.jboss.cache.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:65) at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:119) at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100) at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178) at org.jboss.cache.interceptors.InvocationContextInterceptor.visitPutKeyValueCommand(InvocationContextInterceptor.java:82) at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100) at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287) at org.jboss.cache.invocation.CacheInvocationDelegate.put(CacheInvocationDelegate.java:555) Cache loader and lockParentForChildInsertRemove are prerequisites for this to happen. But the test case (attached, might be a little hard to follow :) ) shows that the deadlock occurs only when adding a node, not when removing.