Hot Rod 2.0 should add error codes for suspected nodes and stopping/stopped caches ---------------------------------------------------------------------------------- Key: ISPN-4717 URL: https://issues.jboss.org/browse/ISPN-4717 Project: Infinispan Issue Type: Bug Affects Versions: 6.0.2.Final Reporter: Galder Zamarreño Assignee: Galder Zamarreño Fix For: 7.0.0.CR1, 7.0.0.Final The way Hot Rod protocol deals with suspected exceptions is hacky. It inspects the error message to detect whether a SuspectException has been passed in. Instead, suspect exceptions should have a dedicated error code so that clients can handle appropriately. On top of that, another exception that should be handled more silently and failover is when a cache is stopping or is stopped. -Currently, this produces the following log messages without affecting functionality- Scrap that, it does get propagated to the client without being able to failover, so it's a bug: {code}2014-09-11 08:11:04,984 ERROR [HotRodDecoder] (HotRodServerWorker-6-1) ISPN005003: Exception reported java.lang.IllegalStateException: Default cache is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:94) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1490) at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:968) at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:960) at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:485) at org.infinispan.server.core.AbstractProtocolDecoder.put(AbstractProtocolDecoder.scala:252) at org.infinispan.server.core.AbstractProtocolDecoder.org$infinispan$server$core$AbstractProtocolDecoder$$decodeValue(AbstractProtocolDecoder.scala:207) at org.infinispan.server.core.AbstractProtocolDecoder.decodeDispatch(AbstractProtocolDecoder.scala:73) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:61) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:362) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149) at org.infinispan.server.core.AbstractProtocolDecoder.channelRead(AbstractProtocolDecoder.scala:471) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:332) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:125) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:744) 2014-09-11 08:11:04,990 ERROR [HotRodDecoder] (HotRodServerWorker-6-1) ISPN005009: Unexpected error before any request parameters read io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.IllegalStateException: Default cache is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:417) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149) at org.infinispan.server.core.AbstractProtocolDecoder.channelRead(AbstractProtocolDecoder.scala:471) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:332) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:125) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:744) Caused by: org.infinispan.server.hotrod.HotRodException: java.lang.IllegalStateException: Default cache is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. at org.infinispan.server.hotrod.HotRodDecoder.createServerException(HotRodDecoder.scala:204) at org.infinispan.server.core.AbstractProtocolDecoder.decodeDispatch(AbstractProtocolDecoder.scala:77) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:61) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:362) ... 12 more Caused by: java.lang.IllegalStateException: Default cache is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:94) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1490) at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:968) at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:960) at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:485) at org.infinispan.server.core.AbstractProtocolDecoder.put(AbstractProtocolDecoder.scala:252) at org.infinispan.server.core.AbstractProtocolDecoder.org$infinispan$server$core$AbstractProtocolDecoder$$decodeValue(AbstractProtocolDecoder.scala:207) at org.infinispan.server.core.AbstractProtocolDecoder.decodeDispatch(AbstractProtocolDecoder.scala:73) ... 14 more 2014-09-11 08:11:04,991 WARN [Codec20] (ForkThread-1,DistTopologyChangeUnderLoadTest) ISPN004005: Error received from the server: io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.IllegalStateException: Default cache is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. {code} Cache stopping/stopped should have a different error code too so that clients can handle it properly.