TransactionTable should ignore view changes during shutdown ----------------------------------------------------------- Key: ISPN-6043 URL: https://issues.jboss.org/browse/ISPN-6043 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 8.1.0.Final Reporter: Dan Berindei Assignee: Dan Berindei Fix For: 8.2.0.Beta1, 8.2.0.Final, 8.1.1.Final During shutdown, {{TransactionTable}} unregisters itself as a view change listener, but it can still receive view change notifications after it stopped the executor service. When that happens, it causes a {{RejectedExecutionException}} that is eventually logged by JGroups: {noformat} pbcast.GMS - JGRP000027: failed passing message up java.lang.RuntimeException: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.util.concurrent.RejectedExecutionException] while invoking method [public void org.infinispan.transaction.TransactionTable.onViewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.transaction.TransactionTable@3d5ab0ba at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:682) at org.jgroups.JChannel.up(JChannel.java:733) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029) at org.jgroups.protocols.RSVP.up(RSVP.java:201) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:394) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732) at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:146) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922) at org.jgroups.stack.Protocol.up(Protocol.java:412) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474) at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982) at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912) at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) at org.jgroups.protocols.FD.up(FD.java:255) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297) at org.jgroups.protocols.MERGE3.up(MERGE3.java:288) at org.jgroups.protocols.Discovery.up(Discovery.java:291) at org.jgroups.protocols.TP.passMessageUp(TP.java:1572) at org.jgroups.protocols.TP$MyHandler.run(TP.java:1791) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.util.concurrent.RejectedExecutionException] while invoking method [public void org.infinispan.transaction.TransactionTable.onViewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.transaction.TransactionTable@3d5ab0ba at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:287) at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22) at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:305) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.notifyViewChange(CacheManagerNotifierImpl.java:88) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$NotifyViewChange.emitNotification(JGroupsTransport.java:638) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.viewAccepted(JGroupsTransport.java:708) at org.jgroups.blocks.MessageDispatcher.handleUpEvent(MessageDispatcher.java:602) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:679) ... 25 more Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1f5986a3 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2e964769[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1696] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325) at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:546) at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:646) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641) at org.infinispan.transaction.TransactionTable.onViewChange(TransactionTable.java:491) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:282) ... 32 more {noformat} The exception is harmless for the stopping cache, the problem is that the following view change listeners are also skipped. We should fix both {{TransactionTable}} to avoid throwing the exception, and {{CacheManagerNotifier}} to ignore any exceptions during view changes.