[jboss-jira] [JBoss JIRA] (WFLY-5128) TimeoutException: Could not acquire lock: Waiting to complete tx

[Michal Vinkler (JIRA)]

Michal Vinkler created WFLY-5128:
------------------------------------

             Summary: TimeoutException: Could not acquire lock: Waiting to complete tx
                 Key: WFLY-5128
                 URL: https://issues.jboss.org/browse/WFLY-5128
             Project: WildFly
          Issue Type: Bug
          Components: Clustering
            Reporter: Michal Vinkler
            Assignee: Paul Ferraro


Seen in these scenarios: 
ejbservlet - shutdown/undeploy
http session - shutdown/undeploy

Setup: 4 node cluster, one node at time is shutdown, while standalone clients keep calling the application.
Approx. 30 seconds after failing one node, one or more nodes intermittently log this error:

{code}
[JBossINF] [0m[31m16:25:28,774 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p3-t13) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on Jc-A2RkwALPqP0gqsUZuC4PWTd_UUdeRtFDD7gv5 on behalf of transaction GlobalTransaction:<perf20>:19672:remote. Waiting to complete tx: RemoteTransaction{modifications=[ReplaceCommand{key=Jc-A2RkwALPqP0gqsUZuC4PWTd_UUdeRtFDD7gv5, oldValue=null, newValue=org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionCacheEntry at 63261b4e, metadata=EmbeddedMetadata{version=null}, flags=[FORCE_WRITE_LOCK, IGNORE_RETURN_VALUES], successful=true, valueMatcher=MATCH_ALWAYS}], lookedUpEntries={Jc-A2RkwALPqP0gqsUZuC4PWTd_UUdeRtFDD7gv5=RepeatableReadEntry(3ac1d699){key=Jc-A2RkwALPqP0gqsUZuC4PWTd_UUdeRtFDD7gv5, value=org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionCacheEntry at 63261b4e, oldValue=null, isCreated=false, isChanged=false, isRemoved=false, isValid=true, skipRemoteGet=false, metadata=EmbeddedMetadata{version=null}}}, lockedKeys=[Jc-A2RkwALPqP0gqsUZuC4PWTd_UUdeRtFDD7gv5], backupKeyLocks=null, lookedUpEntriesTopology=9, isMarkedForRollback=false, tx=GlobalTransaction:<perf20>:18402:remote, state=null}.
[JBossINF] 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.newTimeoutException(AbstractTxLockingInterceptor.java:217)
[JBossINF] 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.waitForTransactionsToComplete(AbstractTxLockingInterceptor.java:209)
[JBossINF] 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:173)
[JBossINF] 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:118)
[JBossINF] 	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:200)
[JBossINF] 	at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
[JBossINF] 	at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] 	at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:141)
[JBossINF] 	at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:199)
[JBossINF] 	at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
[JBossINF] 	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)
[JBossINF] 	at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:196)
[JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:99)
[JBossINF] 	at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
[JBossINF] 	at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:76)
[JBossINF] 	at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] 	at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:129)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
{code}

This error may be the root cause of https://issues.jboss.org/browse/JBEAP-505 (server ends up with Operation ("deploy") failed when restarted after failover).

Server log:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-session-shutdown-dist-sync/4/console-perf21/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)