Hello, I was looking into the many stacktraces being thrown from the testsuite, this one caught my attention (see below). Can't we just discard such errors? if the cache is stopping, or stopped already, we shouldn't really care for invalidations - especially stacktraces and exceptions being returned to the caller. This doesn't solve all the EOF exceptions I'm still experiencing, but it seems to make things a bit better.. I've hacked a solution which implies adding a method: boolean ignoreCommandOnStatus(ComponentStatus status); to the VisitableCommand interface, seems to work fine. Shall I open a JIRA and send a pull request? Details: https://github.com/Sanne/infinispan/commit/ed962ed72bc68765078b6a0f172b95... Cheers, Sanne 2011-07-14 15:16:20,940 ERROR [RebalanceTask] (Rehasher,Infinispan-Cluster,NonStringKeyStateTransferTest-NodeC-7649) ISPN000145: Error during rehash java.lang.IllegalStateException: Default cache is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:83) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:120) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:124) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:177) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274) at org.infinispan.distribution.RebalanceTask.invalidateKeys(RebalanceTask.java:172) at org.infinispan.distribution.RebalanceTask.performRehash(RebalanceTask.java:145) at org.infinispan.distribution.RehashTask.call(RehashTask.java:67) at org.infinispan.distribution.RehashTask.call(RehashTask.java:44) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

TBH I'm not sure why the patch helped with the error messages in the log, I always thought that doing this would just replace the IllegalStateException with an InterruptedException (since do some more stuff after invalidation). Of course the best solution would be to not do rehashing at all when we know we're going to stop the entire cluster anyway (https://issues.jboss.org/browse/ISPN-1239). Dan On Thu, Jul 14, 2011 at 7:48 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > Patch looks good.  Go ahead and issue a pull req. > > On 14 Jul 2011, at 16:26, Sanne Grinovero wrote: > >> Hello, >> I was looking into the many stacktraces being thrown from the >> testsuite, this one caught my attention (see below). >> Can't we just discard such errors? if the cache is stopping, or >> stopped already, we shouldn't really care for invalidations - >> especially stacktraces and exceptions being returned to the caller. >> This doesn't solve all the EOF exceptions I'm still experiencing, but >> it seems to make things a bit better.. I've hacked a solution which >> implies adding a method: >> >> boolean ignoreCommandOnStatus(ComponentStatus status); >> >> to the VisitableCommand interface, seems to work fine. Shall I open a >> JIRA and send a pull request? >> >> Details: >> https://github.com/Sanne/infinispan/commit/ed962ed72bc68765078b6a0f172b95... >> >> Cheers, >> Sanne >> >> 2011-07-14 15:16:20,940 ERROR [RebalanceTask] >> (Rehasher,Infinispan-Cluster,NonStringKeyStateTransferTest-NodeC-7649) >> ISPN000145: Error during rehash >> java.lang.IllegalStateException: Default cache is in 'STOPPING' state >> and this is an invocation not belonging to an on-going transaction, so >> it does not accept new invocations. Either restart it or recreate the >> cache container. >>       at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:83) >>       at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64) >>       at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:120) >>       at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:124) >>       at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:177) >>       at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274) >>       at org.infinispan.distribution.RebalanceTask.invalidateKeys(RebalanceTask.java:172) >>       at org.infinispan.distribution.RebalanceTask.performRehash(RebalanceTask.java:145) >>       at org.infinispan.distribution.RehashTask.call(RehashTask.java:67) >>       at org.infinispan.distribution.RehashTask.call(RehashTask.java:44) >>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>       at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>       at java.lang.Thread.run(Thread.java:662) >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > twitter.com/maniksurtani > > Lead, Infinispan > http://www.infinispan.org > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I was surprised because I would have thought your change would replace each IllegalStateException in the log with an InterruptedException, so I wouldn't have expected the log to be any less noisy. I mean you don't stop the rehashing during invalidation, you just skip invalidation and go to the next phase (notifyCoordinatorPushCompleted), which I thought should throw an InterruptedException. True, we could ignore InterruptedExceptions in RehashTask and not log anything, but based only on this change I was expecting the same number of exceptions in the log. Good to know that I was wrong :) Dan On Fri, Jul 15, 2011 at 1:46 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: > Sorry for not being clear - I'm not assuming that this has anything to > do with the InterruptedException/EOFs being still logged, > just that the output is less noisy if I remove these which where pointless. > > Sanne > > 2011/7/14 Dan Berindei <dan.berindei(a)gmail.com&gt;: >> TBH I'm not sure why the patch helped with the error messages in the >> log, I always thought that doing this would just replace the >> IllegalStateException with an InterruptedException (since do some more >> stuff after invalidation). >> >> Of course the best solution would be to not do rehashing at all when >> we know we're going to stop the entire cluster anyway >> (https://issues.jboss.org/browse/ISPN-1239). >> >> Dan >> >> >> On Thu, Jul 14, 2011 at 7:48 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>> Patch looks good. Go ahead and issue a pull req. >>> >>> On 14 Jul 2011, at 16:26, Sanne Grinovero wrote: >>> >>>> Hello, >>>> I was looking into the many stacktraces being thrown from the >>>> testsuite, this one caught my attention (see below). >>>> Can't we just discard such errors? if the cache is stopping, or >>>> stopped already, we shouldn't really care for invalidations - >>>> especially stacktraces and exceptions being returned to the caller. >>>> This doesn't solve all the EOF exceptions I'm still experiencing, but >>>> it seems to make things a bit better.. I've hacked a solution which >>>> implies adding a method: >>>> >>>> boolean ignoreCommandOnStatus(ComponentStatus status); >>>> >>>> to the VisitableCommand interface, seems to work fine. Shall I open a >>>> JIRA and send a pull request? >>>> >>>> Details: >>>> https://github.com/Sanne/infinispan/commit/ed962ed72bc68765078b6a0f172b95... >>>> >>>> Cheers, >>>> Sanne >>>> >>>> 2011-07-14 15:16:20,940 ERROR [RebalanceTask] >>>> (Rehasher,Infinispan-Cluster,NonStringKeyStateTransferTest-NodeC-7649) >>>> ISPN000145: Error during rehash >>>> java.lang.IllegalStateException: Default cache is in 'STOPPING' state >>>> and this is an invocation not belonging to an on-going transaction, so >>>> it does not accept new invocations. Either restart it or recreate the >>>> cache container. >>>> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:83) >>>> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64) >>>> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:120) >>>> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:124) >>>> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:177) >>>> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274) >>>> at org.infinispan.distribution.RebalanceTask.invalidateKeys(RebalanceTask.java:172) >>>> at org.infinispan.distribution.RebalanceTask.performRehash(RebalanceTask.java:145) >>>> at org.infinispan.distribution.RehashTask.call(RehashTask.java:67) >>>> at org.infinispan.distribution.RehashTask.call(RehashTask.java:44) >>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> twitter.com/maniksurtani >>> >>> Lead, Infinispan >>> http://www.infinispan.org >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev