Caused by: org.infinispan.commons.CacheException: ISPN000332: Remote transaction GlobalTransaction:<InfinispanNode-2277>:5936:remote rolled back because originator is no longer in the cluster ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Key: ISPN-10538 URL: https://issues.jboss.org/browse/ISPN-10538 Project: Infinispan Issue Type: Bug Components: Cross-Site Replication Affects Versions: 7.2.5.Final Environment: Some times at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:256) ~[infinispan-core.jar:7.2.5.Final] is failed Caused by: org.infinispan.commons.CacheException: ISPN000332: Remote transaction GlobalTransaction:<InfinispanNode-2277>:5936:remote rolled back because originator is no longer in the cluster at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:181) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:125) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:113) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85) ~[infinispan-core.jar:7.2.5.Final] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:32) ~[infinispan-core.jar:7.2.5.Final] Reporter: Zhang Lijia Priority: Major Attachments: oss_activity0_0.log_node14, oss_activity0_0.log_node15, oss_trace0_0.log_node14, oss_trace0_0.log_node15 We have two node some times put is failed while some times put is successful. It happens occasionally. The response from other site 2019-08-29-T09:23:23.701+0300 | srnclab18node14 | | OOB-1,InfinispanNode-57675 | TRACE | org.jgroups.protocols.UDP | InfinispanNode-57675: received [dst: InfinispanNode-57675, src: InfinispanNode-2277 (2 headers), size=201 bytes, flags=OOB|DONT_BUNDLE|NO_TOTAL_ORDER], headers are RequestCorrelator: id=200, type=REQ, id=14874, rsp_expected=true, UDP: [cluster_name=InfinispanCluster-common_mediations] 2019-08-29-T09:23:23.702+0300 | srnclab18node14 | | OOB-1,InfinispanNode-57675 | TRACE | org.jgroups.blocks.RequestCorrelator | calling (org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher) with request 14874 2019-08-29-T09:23:23.702+0300 | srnclab18node14 | | OOB-1,InfinispanNode-57675 | TRACE | org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl | Added a new task: 0 task(s) are waiting 2019-08-29-T09:23:23.702+0300 | srnclab18node14 | | remote-thread-InfinispanNode-p3-t1241 | TRACE | org.infinispan.interceptors.locking.PessimisticLockingInterceptor | Locking key operation-228.2710463008511921082.com.nsn.oss.mediation.south.ne3soapcm-228.2710463008511921082.com.nsn.oss.mediation.south.ne3soapcm, no need to check for pending locks. 2019-08-29-T09:23:23.702+0300 | srnclab18node14 | | remote-thread-InfinispanNode-p3-t1241 | TRACE | org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer | Creating and acquiring new lock instance for key operation-228.2710463008511921082.com.nsn.oss.mediation.south.ne3soapcm-228.2710463008511921082.com.nsn.oss.mediation.south.ne3soapcm 2019-08-29-T09:23:23.702+0300 | srnclab18node14 | | remote-thread-InfinispanNode-p3-t1241 | TRACE | org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer | Unlocking lock instance for key operation-228.2710463008511921082.com.nsn.oss.mediation.south.ne3soapcm-228.2710463008511921082.com.nsn.oss.mediation.south.ne3soapcm 2019-08-29-T09:23:23.703+0300 | srnclab18node14 | | remote-thread-InfinispanNode-p3-t1241 | TRACE | org.jgroups.blocks.RequestCorrelator | sending rsp for 14874 to InfinispanNode-2277 while for the successful one 2019-08-29-T08:47:47.555+0300 | srnclab18node14 | | OOB-1,InfinispanNode-57675 | TRACE | org.jgroups.blocks.RequestCorrelator | calling (org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher) with request 14777 2019-08-29-T08:47:47.691+0300 | srnclab18node14 | | remote-thread-InfinispanNode-p3-t1237 | TRACE | org.infinispan.interceptors.locking.PessimisticLockingInterceptor | Locking key operation-224.-8971956864618055540.com.nsn.oss.mediation.south.ne3soapcm-224.-8971956864618055540.com.nsn.oss.mediation.south.ne3soapcm, no need to check for pending locks. 2019-08-29-T08:47:47.691+0300 | srnclab18node14 | | remote-thread-InfinispanNode-p3-t1237 | TRACE | org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer | Creating and acquiring new lock instance for key operation-224.-8971956864618055540.com.nsn.oss.mediation.south.ne3soapcm-224.-8971956864618055540.com.nsn.oss.mediation.south.ne3soapcm 2019-08-29-T08:47:47.691+0300 | srnclab18node14 | | remote-thread-InfinispanNode-p3-t1237 | TRACE | org.jgroups.blocks.RequestCorrelator | sending rsp for 14777 to InfinispanNode-2277 2019-08-29-T08:47:47.691+0300 | srnclab18node14 | | remote-thread-InfinispanNode-p3-t1237 | TRACE | org.jgroups.protocols.UDP | InfinispanNode-57675: sending msg to InfinispanNode-2277, src=InfinispanNode-57675, headers are RequestCorrelator: id=200, type=RSP, id=14777, rsp_expected=false, UDP: [cluster_name=InfinispanCluster-common_mediations]