Could not write method invocation failure due to HeuristicMixedException ------------------------------------------------------------------------ Key: WFLY-7958 URL: https://issues.jboss.org/browse/WFLY-7958 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 10.1.0.Final Reporter: Paul Ferraro Assignee: Paul Ferraro Occured during topology change on *server* during scenarios: eap-7x-failover-ejb-ejbremote-jvmkill-repl-async eap-7x-failover-ejb-ejbremote-shutdown-repl-async eap-7x-failover-ejb-ejbremote-shutdown-repl-sync eap-7x-failover-ejb-ejbremote-undeploy-repl-sync before "Finished local rebalance" INFO message was logged (scenarios eap-7x-failover-ejb-ejbremote-jvmkill-repl-async and eap-7x-failover-ejb-ejbremote-undeploy-repl-sync) or directly after "Received new cluster view" INFO message (scenario eap-7x-failover-ejb-ejbremote-shutdown-repl-async). Server log stacktrace: {code} 16:02:10,148 ERROR [org.jboss.as.ejb3.remote] (default task-111) WFLYEJB0150: Could not write method invocation failure for method public abstract int org.jboss.test.clusterbench.common.ejb.CommonStatefulSB.getSerialAndIncrement() on bean named RemoteStatefulSBImpl for appname clusterbench-ee7 modulename clusterbench-ee7-ejb distinctname due to: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:102) [JBossINF] at org.jboss.as.ejb3.cache.distributable.DistributableCache.getWeakAffinity(DistributableCache.java:69) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.getWeakAffinity(MethodInvocationMessageHandler.java:274) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$600(MethodInvocationMessageHandler.java:68) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:244) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:268) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159) [JBossINF] at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$receiveMessage$2(RemoteConnectionChannel.java:365) [JBossINF] at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:734) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] Caused by: javax.transaction.HeuristicMixedException [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:439) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] ... 13 more [JBossINF] Caused by: javax.transaction.xa.XAException [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:207) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401) [JBossINF] ... 17 more [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf20 [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642) [JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] ... 3 more {code} Link to server log: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failov... Variant with *HeuristicRollbackException* logged on perf20 during scenario eap-7x-failover-ejb-ejbremote-jvmkill-repl-async (2 occurences): {code} 16:08:49,504 ERROR [org.jboss.as.ejb3.remote] (default task-70) WFLYEJB0150: Could not write method invocation failure for method public abstract int org.jboss.test.clusterbench.common.ejb.CommonStatefulSB.getSerialAndIncrement() on bean named RemoteStatefulSBImpl for appname clusterbench-ee7 modulename clusterbench-ee7-ejb distinctname due to: org.infinispan.commons.CacheException: javax.transaction.HeuristicRollbackException [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:102) [JBossINF] at org.jboss.as.ejb3.cache.distributable.DistributableCache.getWeakAffinity(DistributableCache.java:69) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.getWeakAffinity(MethodInvocationMessageHandler.java:274) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$600(MethodInvocationMessageHandler.java:68) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:244) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:268) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76) [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159) [JBossINF] at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$receiveMessage$2(RemoteConnectionChannel.java:365) [JBossINF] at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:734) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] Caused by: javax.transaction.HeuristicRollbackException [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:433) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] ... 13 more [JBossINF] Caused by: javax.transaction.xa.XAException [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114) [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401) [JBossINF] ... 17 more [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 27 [JBossINF] at org.infinispan.statetransfer.StateTransferLockImpl.reportErrorAfterWait(StateTransferLockImpl.java:159) [JBossINF] at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:98) [JBossINF] at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:97) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:249) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:255) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112) [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157) [JBossINF] ... 19 more {code} Link to server log: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failov...