IllegalStateException: channel is not connected exception when shutting down and replication is in progress ----------------------------------------------------------------------------------------------------------- Key: WFLY-5463 URL: https://issues.jboss.org/browse/WFLY-5463 Project: WildFly Issue Type: Bug Components: Clustering Reporter: Richard Janík Assignee: Paul Ferraro Priority: Minor Hi, we're seeing the following exception: {code} ... [JBossINF] 20:12:53,073 WARN [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 79) ISPN000100: Stopping, but there are 272 local transactions and 891 remote transactions that did not finish in time. [JBossINF] 20:12:53,080 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 79) WFLYCLINF0003: Stopped clusterbench-ee7.ear.clusterbench-ee7-web-default.war cache from web container [JBossINF] 20:12:53,097 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 74) ISPN000029: Passivating all entries to disk [JBossINF] 20:12:53,101 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 74) ISPN000030: Passivated 3 entries in 3 milliseconds [JBossINF] 20:12:53,111 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 74) WFLYCLINF0003: Stopped repl cache from web container [JBossINF] 20:12:53,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel web [JBossINF] 20:12:53,163 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel web [JBossINF] 20:12:53,160 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-72) ISPN000073: Unexpected error while replicating: org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:128) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:108) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:545) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:170) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:212) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare(PessimisticLockingInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:128) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitReplaceCommand(AbstractLockingInterceptor.java:66) [JBossINF] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:53) [JBossINF] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:366) [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:235) [JBossINF] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:53) [JBossINF] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:304) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:286) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitReplaceCommand(StateTransferInterceptor.java:128) [JBossINF] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191) [JBossINF] at org.infinispan.interceptors.CacheMgmtInterceptor.visitReplaceCommand(CacheMgmtInterceptor.java:182) [JBossINF] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:53) [JBossINF] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [JBossINF] at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1605) [JBossINF] at org.infinispan.cache.impl.CacheImpl.replaceInternal(CacheImpl.java:1150) [JBossINF] at org.infinispan.cache.impl.CacheImpl.replace(CacheImpl.java:1142) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.replace(DecoratedCache.java:517) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.replace(AbstractDelegatingCache.java:261) [JBossINF] at org.wildfly.clustering.ee.infinispan.CacheEntryMutator.mutate(CacheEntryMutator.java:52) [JBossINF] at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionAttributes.getAttribute(CoarseSessionAttributes.java:62) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.getAttribute(DistributableSession.java:131) [JBossINF] at io.undertow.servlet.spec.HttpSessionImpl.getAttribute(HttpSessionImpl.java:123) [JBossINF] at org.jboss.weld.context.beanstore.http.AbstractSessionBeanStore.getAttribute(AbstractSessionBeanStore.java:91) [JBossINF] at org.jboss.weld.context.beanstore.AttributeBeanStore.get(AttributeBeanStore.java:138) [JBossINF] at org.jboss.weld.context.beanstore.http.AbstractSessionBeanStore.get(AbstractSessionBeanStore.java:79) [JBossINF] at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:83) [JBossINF] at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:114) [JBossINF] at org.jboss.weld.context.PassivatingContextWrapper$AbstractPassivatingContextWrapper.get(PassivatingContextWrapper.java:70) [JBossINF] at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:90) [JBossINF] at org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:165) [JBossINF] at org.jboss.weld.bean.ContextualInstance.getIfExists(ContextualInstance.java:63) [JBossINF] at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:83) [JBossINF] at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125) [JBossINF] at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(Unknown Source) [JBossINF] at org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:38) [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [JBossINF] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) [JBossINF] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [JBossINF] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [JBossINF] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [JBossINF] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [JBossINF] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [JBossINF] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72) [JBossINF] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [JBossINF] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:778) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] Caused by: java.lang.IllegalStateException: channel is not connected [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:692) [JBossINF] at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:175) [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:360) [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:103) [JBossINF] at org.jgroups.blocks.Request.execute(Request.java:83) [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:367) [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:373) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:300) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:104) [JBossINF] ... 85 more ... {code} It gets thrown when the server is shutting down and it seems that some replication is still in progress. The exception makes perfect sense due to the state in which the server is, however the ERROR log level seems high. Would it make sense to lower it? (INFO || WARN || DEBUG)