TimeoutException: Replication timeout when handling request with DIST cache --------------------------------------------------------------------------- Key: ISPN-6993 URL: https://issues.jboss.org/browse/ISPN-6993 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 8.2.4.Final Environment: two data owners ,three node Reporter: yingming jiang Priority: Blocker 8.2.4 distributed-cache when the node members more than data owners ,raise time out exception my config as follow: infinispan.xml <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:infinispan:config:8.2 http://www.infinispan.org/schemas/infinispan-config-8.2.xsd" xmlns="urn:infinispan:config:8.2"> <jgroups> <stack-file name="tcp" path="jgroups-my.xml" /> </jgroups> <cache-container default-cache="default"> <transport stack="tcp" /> <distributed-cache name="links"> </distributed-cache> </cache-container> </infinispan> jgroups-my.xml <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-3.6.xsd"> <TCP bind_addr="${jgroups.tcp.address:127.0.0.1}" bind_port="${jgroups.tcp.port:7800}" recv_buf_size="${tcp.recv_buf_size:20M}" send_buf_size="${tcp.send_buf_size:2M}" sock_conn_timeout="300"/> <TCPGOSSIP initial_hosts="${jgroups.tcpgossip.initial_hosts}"/> <MERGE3/> <FD/> <VERIFY_SUSPECT/> <pbcast.NAKACK2 use_mcast_xmit="false"/> <UNICAST3/> <pbcast.STABLE/> <pbcast.GMS/> <MFC/> <FRAG2/> </config> error message: 2016-09-01 11:16:35,380 ERROR [InvocationContextInterceptor] (main) ISPN000136: Error executing command GetKeyValueCommand, writing keys [] org.infinispan.util.concurrent.TimeoutException: Replication timeout for sptn-win-63-48742 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.staggeredProcessNext(CommandAwareRpcDispatcher.java:375) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.lambda$processCallsStaggered$3(CommandAwareRpcDispatcher.java:357) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Exception in thread "main" org.infinispan.util.concurrent.TimeoutException: Replication timeout for sptn-win-63-48742 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.staggeredProcessNext(CommandAwareRpcDispatcher.java:375) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.lambda$processCallsStaggered$3(CommandAwareRpcDispatcher.java:357) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) the bug is the same as follow ,the issue is gone when using -Dinfinispan.stagger.delay=0 https://issues.jboss.org/browse/WFLY-6926