[infinispan-issues] [JBoss JIRA] (ISPN-1754) IllegalStateException during cluster shutdown

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660290#comment-12660290 ] 

Dan Berindei commented on ISPN-1754:
------------------------------------

Maybe Mircea can explain the details better, but I don't think there's anything extraordinary going on here.

The server was shut down, and al the threads that were processing requests were interrupted (I assume with ThreadPoolExecutor.shutdownNow()). This caused the put operation's implicit transaction to fail, and the transaction manager tried to execute a rollback command on the cache while it was shutting down - hence the error message.

I think there may be a problem in that the put command is supposed to tell the transaction manager that it doesn't need to call rollback, but that's not very serious - it would be ignored if the cache wasn't shutting down.

Either way, the HotRod client should retry the operation on another server and the user shouldn't notice anything.
                
> IllegalStateException during cluster shutdown
> ---------------------------------------------
>
>                 Key: ISPN-1754
>                 URL: https://issues.jboss.org/browse/ISPN-1754
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Distributed Cache
>    Affects Versions: 5.1.0.CR4
>            Reporter: Michal Linhard
>            Assignee: Manik Surtani
>
> I get these during stress/failover test while shutting down the servers
> {code}
> 05:44:54,051 ERROR [org.infinispan.transaction.TransactionCoordinator] ISPN000098: Exception while rollback: java.lang.IllegalStateException: Cache 'testCache' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so i..
> 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:100) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:54) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:208) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:176) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:89) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:117)
> 	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:403)
> 	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:104)
> 	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159)
> 	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1162)
> 	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119)
> 	at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
> 	at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:951) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.CacheImpl.put(CacheImpl.java:654) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.DecoratedCache.put(DecoratedCache.java:179) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:119) [infinispan-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.server.core.AbstractProtocolDecoder.put(AbstractProtocolDecoder.scala:187) [infinispan-server-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.server.core.AbstractProtocolDecoder.decodeValue(AbstractProtocolDecoder.scala:141) [infinispan-server-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:71) [infinispan-server-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:44) [infinispan-server-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.jboss.netty.handler.codec.replay.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250) [infinispan-server-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.jboss.netty.handler.codec.replay.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223) [infinispan-server-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:351) [infinispan-server-core-5.1.0.CR4.jar:5.1.0.CR4]
> 	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.2.6.Final.jar:]
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) [netty-3.2.6.Final.jar:]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_29]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_29]
> 	at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira