May 2018 - jboss-jira - Jboss List Archives

[JBoss JIRA] (WFLY-10317) CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException in a 4 node cluster during fail-over

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10317?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10317: ----------------------------------- Description: We observed this new "CIRCULAR REFERENCE" message in the logs while running one of the regular fail-over tests. Given that performance is not impacted, the goal of this Issue is to get some investigation on what changed and why this message appeared. We observed the following error when the cache is configured as *replicated* and synchronous. We did *not* have the error when the cache is configured as *distributed* and synchronous. The following Jenkins runs provide both the new and the old situation for comparison: * [CIRCULAR REFERENCE|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/tborga...] : jenkins execution with latest wildfly version 13.0.0.Alpha1-SNAPSHOT where the "CIRCULAR REFERENCE" message can be seen in logs * [NO CIRCULAR REFERENCE|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/tborga...] : jenkins execution with latest wildfly version 12.0.0.Final where no "CIRCULAR REFERENCE" message is present In the following we give more details about the two runs. For the first run, we used master branch from [13.0.0.Alpha1-SNAPSHOT|https://github.com/wildfly/wildfly/tree/master]. We observe lot of errors (more than a hundred on each node) like the following, note they occur right after one node has been stopped and restarted and then another node is stopped and restarted: {noformat} 2018/05/02 09:03:33:843 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-5) New session created: TXIN2deAqLp1QHmZzzQXXrUyTslOYCAx9pazOIo5 [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-6) New session created: px4vAOKxK8pJ74234T406YgN3jG4e0Z72MDNEkZS [JBossINF] [0m[0m09:03:34,009 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-20) New session created: LzmtOORpjJqz7ZqRhmIg6rro3Top9GlWBixfcS7D [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-1) New session created: FYJS0bQVNA4bxHusRV2gI6stJCQxO0rJ2IuzA7b4 [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-9) New session created: Jh5jc4oR_KeSt29lfasKTdszo4dWrr2_A8_nxm3x [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-2) New session created: 5AVElPWyS2H2WvqhfdXuZDOltSvQLyodrR1ckK5S [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-3) New session created: _selI14cpR26Q_zCUFCj5Vq_-B2YkPIsi6BB_kMv [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-7) New session created: DU8zan7woAW5gNRX_JraITeBnnXOTHla2q_5DOpZ [JBossINF] [0m[0m09:03:34,011 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-8) New session created: uU7mbyQQNWsdhKR091t66AZAQ089IFlYuCqZR0ml [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-4) New session created: Tm9d19lL4oFFotC_2PETa1WRKR3kypz3VEhrYF1f [JBossINF] [0m[0m09:04:35,431 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] [0m[0m09:04:35,437 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m09:05:43,742 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] [0m[0m09:05:43,744 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[31m09:05:53,824 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAttributesKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAccessMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21248 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[31m09:05:53,840 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAccessMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAttributesKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[31m09:05:53,843 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] ... 1 more [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] ... 1 more [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19] [JBossINF] {noformat} For the second run, we used tag [12.0.0.Final|https://github.com/wildfly/wildfly/tree/12.0.0.Final] and we did not have the CIRCULAR REFERENCE message; anyway we observed a high number (over a hundred for each node) of the following exceptions, note they occur (just like before) right after one node has been stopped and restarted and then another node is stopped and restarted: {noformat} 2018/05/02 11:17:02:743 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! [JBossINF] [0m[0m11:18:06,946 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] [JBossINF] [0m[0m11:18:06,957 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m11:18:06,960 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] [JBossINF] [0m[0m11:18:06,962 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m11:19:18,519 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] [JBossINF] [0m[0m11:19:18,522 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m11:19:18,523 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[31m11:19:28,697 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAccessMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionAttributesKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionCreationMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAttributesKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAccessMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21977 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-14) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] ... 1 more {noformat} was: During one of the regular fail-over tests, we observed the following error when the cache is configured as replicated and synchronous. We did not have the error when the cache is configured as distributed and synchronous. We used master branch from [13.0.0.Alpha1-SNAPSHOT|https://github.com/wildfly/wildfly/tree/master]. We observe lot of errors (more than a hundred on each node) like the following, note they occur right after one node has been stopped and restarted and then another node is stopped and restarted: {noformat} 2018/05/02 09:03:33:843 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-5) New session created: TXIN2deAqLp1QHmZzzQXXrUyTslOYCAx9pazOIo5 [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-6) New session created: px4vAOKxK8pJ74234T406YgN3jG4e0Z72MDNEkZS [JBossINF] [0m[0m09:03:34,009 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-20) New session created: LzmtOORpjJqz7ZqRhmIg6rro3Top9GlWBixfcS7D [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-1) New session created: FYJS0bQVNA4bxHusRV2gI6stJCQxO0rJ2IuzA7b4 [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-9) New session created: Jh5jc4oR_KeSt29lfasKTdszo4dWrr2_A8_nxm3x [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-2) New session created: 5AVElPWyS2H2WvqhfdXuZDOltSvQLyodrR1ckK5S [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-3) New session created: _selI14cpR26Q_zCUFCj5Vq_-B2YkPIsi6BB_kMv [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-7) New session created: DU8zan7woAW5gNRX_JraITeBnnXOTHla2q_5DOpZ [JBossINF] [0m[0m09:03:34,011 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-8) New session created: uU7mbyQQNWsdhKR091t66AZAQ089IFlYuCqZR0ml [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-4) New session created: Tm9d19lL4oFFotC_2PETa1WRKR3kypz3VEhrYF1f [JBossINF] [0m[0m09:04:35,431 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] [0m[0m09:04:35,437 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m09:05:43,742 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] [0m[0m09:05:43,744 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[31m09:05:53,824 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAttributesKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAccessMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21248 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[31m09:05:53,840 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAccessMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAttributesKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[31m09:05:53,843 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] ... 1 more [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] ... 1 more [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19] [JBossINF] {noformat} We tried also with tag [12.0.0.Final|https://github.com/wildfly/wildfly/tree/12.0.0.Final] and we did not have CIRCULAR REFERENCE; anyway we observed a high number (over a hundred for each node) of the following exceptions, note they occur (just like before) right after one node has been stopped and restarted and then another node is stopped and restarted: {noformat} 2018/05/02 11:17:02:743 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! [JBossINF] [0m[0m11:18:06,946 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] [JBossINF] [0m[0m11:18:06,957 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m11:18:06,960 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] [JBossINF] [0m[0m11:18:06,962 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] [0m[0m11:19:18,519 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] [JBossINF] [0m[0m11:19:18,522 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m11:19:18,523 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] [0m[31m11:19:28,697 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAccessMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionAttributesKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionCreationMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAttributesKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAccessMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21977 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-14) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] ... 1 more {noformat} > CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException in a 4 node cluster during fail-over > ------------------------------------------------------------------------------------------------------- > > Key: WFLY-10317 > URL: https://issues.jboss.org/browse/WFLY-10317 > Project: WildFly > Issue Type: Bug > Components: Clustering > Affects Versions: 13.0.0.Beta1 > Reporter: tommaso borgato > Assignee: Paul Ferraro > Priority: Minor > Attachments: clusterbench-ee7.ear > > > We observed this new "CIRCULAR REFERENCE" message in the logs while running one of the regular fail-over tests. > Given that performance is not impacted, the goal of this Issue is to get some investigation on what changed and why this message appeared. > We observed the following error when the cache is configured as *replicated* and synchronous. > We did *not* have the error when the cache is configured as *distributed* and synchronous. > The following Jenkins runs provide both the new and the old situation for comparison: > * [CIRCULAR REFERENCE|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/tborga...] : jenkins execution with latest wildfly version 13.0.0.Alpha1-SNAPSHOT where the "CIRCULAR REFERENCE" message can be seen in logs > * [NO CIRCULAR REFERENCE|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/tborga...] : jenkins execution with latest wildfly version 12.0.0.Final where no "CIRCULAR REFERENCE" message is present > In the following we give more details about the two runs. > For the first run, we used master branch from [13.0.0.Alpha1-SNAPSHOT|https://github.com/wildfly/wildfly/tree/master]. > We observe lot of errors (more than a hundred on each node) like the following, note they occur right after one node has been stopped and restarted and then another node is stopped and restarted: > {noformat} > 2018/05/02 09:03:33:843 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! > [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-5) New session created: TXIN2deAqLp1QHmZzzQXXrUyTslOYCAx9pazOIo5 > [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-6) New session created: px4vAOKxK8pJ74234T406YgN3jG4e0Z72MDNEkZS > [JBossINF] [0m[0m09:03:34,009 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-20) New session created: LzmtOORpjJqz7ZqRhmIg6rro3Top9GlWBixfcS7D > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-1) New session created: FYJS0bQVNA4bxHusRV2gI6stJCQxO0rJ2IuzA7b4 > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-9) New session created: Jh5jc4oR_KeSt29lfasKTdszo4dWrr2_A8_nxm3x > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-2) New session created: 5AVElPWyS2H2WvqhfdXuZDOltSvQLyodrR1ckK5S > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-3) New session created: _selI14cpR26Q_zCUFCj5Vq_-B2YkPIsi6BB_kMv > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-7) New session created: DU8zan7woAW5gNRX_JraITeBnnXOTHla2q_5DOpZ > [JBossINF] [0m[0m09:03:34,011 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-8) New session created: uU7mbyQQNWsdhKR091t66AZAQ089IFlYuCqZR0ml > [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-4) New session created: Tm9d19lL4oFFotC_2PETa1WRKR3kypz3VEhrYF1f > [JBossINF] [0m[0m09:04:35,431 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] > [JBossINF] [0m[0m09:04:35,437 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] > [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] > [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] > [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m09:05:43,742 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] > [JBossINF] [0m[0m09:05:43,744 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] > [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] > [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] > [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[31m09:05:53,824 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAttributesKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAccessMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21248 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[31m09:05:53,840 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAccessMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAttributesKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[31m09:05:53,843 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 > [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) > [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) > [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122) > [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] ... 1 more > [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 > [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) > [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37) > [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) > [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) > [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122) > [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) > [JBossINF] ... 1 more > [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19] > [JBossINF] > {noformat} > For the second run, we used tag [12.0.0.Final|https://github.com/wildfly/wildfly/tree/12.0.0.Final] and we did not have the CIRCULAR REFERENCE message; anyway we observed a high number (over a hundred for each node) of the following exceptions, note they occur (just like before) right after one node has been stopped and restarted and then another node is stopped and restarted: > {noformat} > 2018/05/02 11:17:02:743 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! > [JBossINF] [0m[0m11:18:06,946 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] > [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] > [JBossINF] [0m[0m11:18:06,957 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m11:18:06,960 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] > [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] > [JBossINF] [0m[0m11:18:06,962 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m11:19:18,519 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] > [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] > [JBossINF] [0m[0m11:19:18,522 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m11:19:18,523 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] > [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] > [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[31m11:19:28,697 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAccessMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionAttributesKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionCreationMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAttributesKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAccessMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21977 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-14) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 > [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) > [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166) > [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126) > [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (SWSQE-183) Validate that Kiali was successfully deployed

by Matt Mahoney (JIRA)

[ https://issues.jboss.org/browse/SWSQE-183?page=com.atlassian.jira.plugin.... ] Matt Mahoney reassigned SWSQE-183: ---------------------------------- Assignee: Sunil kondkar (was: Michael Foley) > Validate that Kiali was successfully deployed > --------------------------------------------- > > Key: SWSQE-183 > URL: https://issues.jboss.org/browse/SWSQE-183 > Project: Kiali QE > Issue Type: Sub-task > Reporter: Matt Mahoney > Assignee: Sunil kondkar > -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10317) CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException in a 4 node cluster during fail-over

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10317?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10317: ----------------------------------- Priority: Minor (was: Major) > CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException in a 4 node cluster during fail-over > ------------------------------------------------------------------------------------------------------- > > Key: WFLY-10317 > URL: https://issues.jboss.org/browse/WFLY-10317 > Project: WildFly > Issue Type: Bug > Components: Clustering > Affects Versions: 13.0.0.Beta1 > Reporter: tommaso borgato > Assignee: Paul Ferraro > Priority: Minor > Attachments: clusterbench-ee7.ear > > > During one of the regular fail-over tests, we observed the following error when the cache is configured as replicated and synchronous. > We did not have the error when the cache is configured as distributed and synchronous. > We used master branch from [13.0.0.Alpha1-SNAPSHOT|https://github.com/wildfly/wildfly/tree/master]. > We observe lot of errors (more than a hundred on each node) like the following, note they occur right after one node has been stopped and restarted and then another node is stopped and restarted: > {noformat} > 2018/05/02 09:03:33:843 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! > [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-5) New session created: TXIN2deAqLp1QHmZzzQXXrUyTslOYCAx9pazOIo5 > [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-6) New session created: px4vAOKxK8pJ74234T406YgN3jG4e0Z72MDNEkZS > [JBossINF] [0m[0m09:03:34,009 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-20) New session created: LzmtOORpjJqz7ZqRhmIg6rro3Top9GlWBixfcS7D > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-1) New session created: FYJS0bQVNA4bxHusRV2gI6stJCQxO0rJ2IuzA7b4 > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-9) New session created: Jh5jc4oR_KeSt29lfasKTdszo4dWrr2_A8_nxm3x > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-2) New session created: 5AVElPWyS2H2WvqhfdXuZDOltSvQLyodrR1ckK5S > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-3) New session created: _selI14cpR26Q_zCUFCj5Vq_-B2YkPIsi6BB_kMv > [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-7) New session created: DU8zan7woAW5gNRX_JraITeBnnXOTHla2q_5DOpZ > [JBossINF] [0m[0m09:03:34,011 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-8) New session created: uU7mbyQQNWsdhKR091t66AZAQ089IFlYuCqZR0ml > [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-4) New session created: Tm9d19lL4oFFotC_2PETa1WRKR3kypz3VEhrYF1f > [JBossINF] [0m[0m09:04:35,431 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] > [JBossINF] [0m[0m09:04:35,437 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] > [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] > [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] > [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m09:05:43,742 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] > [JBossINF] [0m[0m09:05:43,744 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] > [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] > [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] > [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[31m09:05:53,824 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAttributesKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAccessMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21248 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[31m09:05:53,840 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAccessMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAttributesKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[31m09:05:53,843 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 > [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) > [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) > [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122) > [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] ... 1 more > [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19 > [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) > [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37) > [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) > [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) > [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122) > [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) > [JBossINF] ... 1 more > [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19] > [JBossINF] > {noformat} > We tried also with tag [12.0.0.Final|https://github.com/wildfly/wildfly/tree/12.0.0.Final] and we did not have CIRCULAR REFERENCE; anyway we observed a high number (over a hundred for each node) of the following exceptions, note they occur (just like before) right after one node has been stopped and restarted and then another node is stopped and restarted: > {noformat} > 2018/05/02 11:17:02:743 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started! > [JBossINF] [0m[0m11:18:06,946 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] > [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] > [JBossINF] [0m[0m11:18:06,957 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m11:18:06,960 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] > [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18] > [JBossINF] [0m[0m11:18:06,962 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster > [JBossINF] [0m[0m11:19:18,519 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] > [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] > [JBossINF] [0m[0m11:19:18,522 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m11:19:18,523 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] > [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19] > [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster > [JBossINF] [0m[31m11:19:28,697 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAccessMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionAttributesKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionCreationMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAttributesKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAccessMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21977 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-14) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 > [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) > [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166) > [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126) > [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19 > [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) > [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFWIP-34) The right bracket in prompt of workflow mode should be colorized by default color

by Jean-Francois Denise (JIRA)

[ https://issues.jboss.org/browse/WFWIP-34?page=com.atlassian.jira.plugin.s... ] Jean-Francois Denise commented on WFWIP-34: ------------------------------------------- yep, strange to have end not the same color as start. Would vote for a bug now seeing it. > The right bracket in prompt of workflow mode should be colorized by default color > --------------------------------------------------------------------------------- > > Key: WFWIP-34 > URL: https://issues.jboss.org/browse/WFWIP-34 > Project: WildFly WIP > Issue Type: Bug > Reporter: Erich Duda > Assignee: Ingo Weiss > Labels: WFCORE-3577 > Attachments: screesnhot.png > > > !screesnhot.png|thumbnail! > As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be colored by default color as the left bracket. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFWIP-34) The right bracket in prompt of workflow mode should be colorized by default color

by Rostislav Svoboda (JIRA)

[ https://issues.jboss.org/browse/WFWIP-34?page=com.atlassian.jira.plugin.s... ] Rostislav Svoboda commented on WFWIP-34: ---------------------------------------- I thought it was a bug too, I expected the right bracket to be in default color. We can always ask [~jdenise] ;) > The right bracket in prompt of workflow mode should be colorized by default color > --------------------------------------------------------------------------------- > > Key: WFWIP-34 > URL: https://issues.jboss.org/browse/WFWIP-34 > Project: WildFly WIP > Issue Type: Bug > Reporter: Erich Duda > Assignee: Ingo Weiss > Labels: WFCORE-3577 > Attachments: screesnhot.png > > > !screesnhot.png|thumbnail! > As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be colored by default color as the left bracket. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFWIP-34) The right bracket in prompt of workflow mode should be colorized by default color

by Erich Duda (JIRA)

[ https://issues.jboss.org/browse/WFWIP-34?page=com.atlassian.jira.plugin.s... ] Erich Duda commented on WFWIP-34: --------------------------------- Everyone on presentation told me that it is a bug. TBH I don't have strong opinion what is better. > The right bracket in prompt of workflow mode should be colorized by default color > --------------------------------------------------------------------------------- > > Key: WFWIP-34 > URL: https://issues.jboss.org/browse/WFWIP-34 > Project: WildFly WIP > Issue Type: Bug > Reporter: Erich Duda > Assignee: Ingo Weiss > Labels: WFCORE-3577 > Attachments: screesnhot.png > > > !screesnhot.png|thumbnail! > As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be colored by default color as the left bracket. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFWIP-34) The right bracket in prompt of workflow mode should be colorized by default color

by Ingo Weiss (JIRA)

[ https://issues.jboss.org/browse/WFWIP-34?page=com.atlassian.jira.plugin.s... ] Ingo Weiss commented on WFWIP-34: --------------------------------- I had actually done this on purpose. Do you think it looks too bad? > The right bracket in prompt of workflow mode should be colorized by default color > --------------------------------------------------------------------------------- > > Key: WFWIP-34 > URL: https://issues.jboss.org/browse/WFWIP-34 > Project: WildFly WIP > Issue Type: Bug > Reporter: Erich Duda > Assignee: Ingo Weiss > Labels: WFCORE-3577 > Attachments: screesnhot.png > > > !screesnhot.png|thumbnail! > As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be colored by default color as the left bracket. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFWIP-34) The right bracket in prompt of workflow mode should be colorized by prompt color

by Erich Duda (JIRA)

[ https://issues.jboss.org/browse/WFWIP-34?page=com.atlassian.jira.plugin.s... ] Erich Duda updated WFWIP-34: ---------------------------- Description: !screesnhot.png|thumbnail! As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be colored by default color as the left bracket. was: !screesnhot.png|thumbnail! As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be blue ({{prompt-color}}). > The right bracket in prompt of workflow mode should be colorized by prompt color > -------------------------------------------------------------------------------- > > Key: WFWIP-34 > URL: https://issues.jboss.org/browse/WFWIP-34 > Project: WildFly WIP > Issue Type: Bug > Reporter: Erich Duda > Assignee: Ingo Weiss > Labels: WFCORE-3577 > Attachments: screesnhot.png > > > !screesnhot.png|thumbnail! > As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be colored by default color as the left bracket. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFWIP-34) The right bracket in prompt of workflow mode should be colorized by default color

by Erich Duda (JIRA)

[ https://issues.jboss.org/browse/WFWIP-34?page=com.atlassian.jira.plugin.s... ] Erich Duda updated WFWIP-34: ---------------------------- Summary: The right bracket in prompt of workflow mode should be colorized by default color (was: The right bracket in prompt of workflow mode should be colorized by prompt color) > The right bracket in prompt of workflow mode should be colorized by default color > --------------------------------------------------------------------------------- > > Key: WFWIP-34 > URL: https://issues.jboss.org/browse/WFWIP-34 > Project: WildFly WIP > Issue Type: Bug > Reporter: Erich Duda > Assignee: Ingo Weiss > Labels: WFCORE-3577 > Attachments: screesnhot.png > > > !screesnhot.png|thumbnail! > As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be colored by default color as the left bracket. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFWIP-34) The right bracket in prompt of workflow mode should be colorized by prompt color

by Erich Duda (JIRA)

Erich Duda created WFWIP-34: ------------------------------- Summary: The right bracket in prompt of workflow mode should be colorized by prompt color Key: WFWIP-34 URL: https://issues.jboss.org/browse/WFWIP-34 Project: WildFly WIP Issue Type: Bug Reporter: Erich Duda Assignee: Ingo Weiss Attachments: screesnhot.png !screesnhot.png|thumbnail! As you can see on the screenshot, the right bracket in prompt is green what is {{workflow-color}} but it should be blue ({{prompt-color}}). -- This message was sent by Atlassian JIRA (v7.5.0#75005)

8 years, 2 months

1
0
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

jboss-jira May 2018