Error executing command PrepareCommand/GetKeyValueCommand due to replication timeout ------------------------------------------------------------------------------------ Key: WFLY-6355 URL: https://issues.jboss.org/browse/WFLY-6355 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 10.0.0.Final Reporter: Michal Vinkler Assignee: Paul Ferraro This is a separate issue originally logged as part of JBEAP-794 (#2 in this [comment|https://issues.jboss.org/browse/JBEAP-794?focusedCommentId=131706...]). It comes with 2 different messages: {code:title=1. Error executing command GetKeyValueCommand} [JBossINF] 03:23:38,763 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-92) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf19 [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$175(JGroupsTransport.java:612) [JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) {code} This one appears exactly in the same scenarios and numbers as JBEAP-3696 - see [occurrences report|http://download.eng.brq.redhat.com/scratch/mvinkler/reports/occurr...] (ejb-ejbservlet and http-session scenarios with *REPL* cache) {code:title=2. Error executing command PrepareCommand} [JBossINF] [0m[31m03:23:38,454 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-24) ISPN000136: Error executing command PrepareCommand, writing keys [UOzA9Tq6F40Y0pcaLUpLe6YLgsYPuFE3SZQoH7iW, UOzA9Tq6F40Y0pcaLUpLe6YLgsYPuFE3SZQoH7iW, UOzA9Tq6F40Y0pcaLUpLe6YLgsYPuFE3SZQoH7iW]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf19 [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$175(JGroupsTransport.java:612) [JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) {code} This one appears exactly in the same scenarios and numbers as JBEAP-3780, JBEAP-3781 (and partially also JBEAP-3782) - see [occurrences report|http://download.eng.brq.redhat.com/scratch/mvinkler/reports/occurr...] (ejb-remote, ejb-ejbservlet and http-session scenarios with *REPL* cache and *SYNC* replication) Server link: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-... Occurrences report: http://download.eng.brq.redhat.com/scratch/mvinkler/reports/occurrences/E...