[stderr] CacheException: javax.transaction.HeuristicRollbackException at server shutdown ---------------------------------------------------------------------------------------- Key: ISPN-5610 URL: https://issues.jboss.org/browse/ISPN-5610 Project: Infinispan Issue Type: Bug Affects Versions: 7.2.3.Final Reporter: Michal Vinkler Assignee: Paul Ferraro Seen in our failover tests for remote stateful EJBs during server shutdown. This error is intermittent and was sometimes seen on one node in 4-nodes cluster. When server is shutting down, this [stderr] output occurs (among other errors) in the server log: {code:} [JBossINF] [0m[0m13:04:52,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0008: Undertow HTTP listener default suspending [JBossINF] [0m[31m13:04:52,865 ERROR [stderr] (ServerService Thread Pool -- 84) Exception in thread "ServerService Thread Pool -- 84" org.infinispan.commons.CacheException: javax.transaction.HeuristicRollbackException [JBossINF] [0m[0m13:04:52,865 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0007: Undertow HTTP listener default stopped, was bound to perf19/10.16.90.56:8080 [JBossINF] [0m[31m13:04:52,865 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:50) [JBossINF] [0m[31m13:04:52,865 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.CacheRegistry.close(CacheRegistry.java:87) [JBossINF] [0m[31m13:04:52,865 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.CacheRegistryFactory$1.close(CacheRegistryFactory.java:56) [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.RegistryBuilder.stop(RegistryBuilder.java:88) [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.service.AsynchronousServiceBuilder$2.run(AsynchronousServiceBuilder.java:130) [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at java.lang.Thread.run(Thread.java:745) [JBossINF] [0m[0m13:04:52,866 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0004: Undertow 1.3.0.Beta3 stopping [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at org.jboss.threads.JBossThread.run(JBossThread.java:320) [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: javax.transaction.HeuristicRollbackException [JBossINF] [0m[31m13:04:52,867 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:398) [JBossINF] [0m[31m13:04:52,867 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:413) [JBossINF] [0m[0m13:04:52,868 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 88) ISPN000029: Passivating all entries to disk [JBossINF] [0m[31m13:04:52,870 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:303) [JBossINF] [0m[31m13:04:52,872 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:104) [JBossINF] [0m[0m13:04:52,872 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 88) ISPN000030: Passivated 2 entries in 3 milliseconds [JBossINF] [0m[31m13:04:52,872 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73) [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48) [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) ... 8 more [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: javax.transaction.xa.XAException [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213) [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159) [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112) [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:367) [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) ... 13 more [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46) [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:71) [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:586) [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287) [JBossINF] [0m[31m13:04:52,875 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:222) [JBossINF] [0m[31m13:04:52,875 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:206) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:102) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:96) [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:99) [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36) [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:138) [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:125) [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42) [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:200) [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:88) [JBossINF] [0m[31m13:04:52,880 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157) [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) ... 15 more [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.NullPointerException] while invoking method [public void org.wildfly.clustering.server.registry.CacheRegistry.removed(org.infinispan.notifications.cachelistener.event.CacheEntryRemovedEvent)] on listener instance: org.wildfly.clustering.server.registry.CacheRegistryFactory$1@76942caa [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:291) [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22) [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309) [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1212) [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1207) [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1184) [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryRemoved(CacheNotifierImpl.java:356) [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:132) [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.commitSingleEntry(ClusteringDependentLogic.java:493) [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:108) [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:371) [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:549) [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:357) [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:104) [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:96) [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:99) [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36) [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:138) [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:125) [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,887 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,887 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [JBossINF] [0m[31m13:04:52,887 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39) [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:200) [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:88) [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) [JBossINF] [0m[0m13:04:52,890 INFO [org.infinispan.CLUSTER] (ServerService Thread Pool -- 88) ISPN000310: Starting cluster-wide rebalance for cache dist, topology CacheTopology{id=37, rebalanceId=15, currentCH=DefaultConsistentHash{ns=80, owners = (2)[perf20: 39+15, perf21: 41+12]}, pendingCH=DefaultConsistentHash{ns=80, owners = (2)[perf20: 40+40, perf21: 40+40]}, unionCH=null, actualMembers=[perf20, perf21]} [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:113) [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85) [JBossINF] [0m[0m13:04:52,890 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 88) WFLYCLINF0003: Stopped dist cache from ejb container [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:32) [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at java.lang.Thread.run(Thread.java:745) [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: java.lang.NullPointerException [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.CacheRegistry.notifyListeners(CacheRegistry.java:183) [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.CacheRegistry.removed(CacheRegistry.java:178) [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) at java.lang.reflect.Method.invoke(Method.java:497) [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286) [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) ... 54 more [JBossINF] [0m[0m13:04:52,902 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t30) ISPN000336: Finished cluster-wide rebalance for cache dist, topology id = 37 [JBossINF] [0m[0m13:04:52,904 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting JGroups channel ejb [JBossINF] [0m[0m13:04:52,905 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher for channel ejb [JBossINF] [0m[0m13:04:52,957 INFO [org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: WildFly Core 2.0.0.Alpha9 "Kenny" stopped in 13744ms {code} RA: I believe the problem here is that ActiveTransactionBatch is throwing a raw new CacheException(e). We should have a logger defined for the ee subsystem so that it throws something like this: InfinispanEELogger.ROOT_LOGGER.debug(...)); At the moment, such a logger is not defined. Server log: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e... http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...