July 2018 - jboss-jira - Jboss List Archives

[JBoss JIRA] (WFLY-10763) ISPN000476: Timed out waiting for responses for request

by tommaso borgato (JIRA)

tommaso borgato created WFLY-10763: -------------------------------------- Summary: ISPN000476: Timed out waiting for responses for request Key: WFLY-10763 URL: https://issues.jboss.org/browse/WFLY-10763 Project: WildFly Issue Type: Bug Components: Clustering Reporter: tommaso borgato Assignee: Paul Ferraro The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. The error was observed on node [dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...] : {noformat} [JBossINF] [0m[31m04:10:27,889 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-178) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 40700 from dev213 [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:127) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 40700 from dev213 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] ... 1 more [JBossINF] Suppressed: org.infinispan.util.logging.TraceException [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:127) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] ... 1 more {noformat} {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} {quote} Please also note that this errors was observed *right after* other error described in [WFLY-10758|https://issues.jboss.org/browse/WFLY-10758] so they may be related. Please refer to that issue for the sequence of events that led to the error. {quote} -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10762) ISPN000136: Error executing command GetKeyValueCommand

by tommaso borgato (JIRA)

tommaso borgato created WFLY-10762: -------------------------------------- Summary: ISPN000136: Error executing command GetKeyValueCommand Key: WFLY-10762 URL: https://issues.jboss.org/browse/WFLY-10762 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 14.0.0.CR1 Reporter: tommaso borgato Assignee: Paul Ferraro The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. The error was observed on node [dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...] : {noformat} [JBossINF] [0m[31m04:10:27,872 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 40684 from dev213 [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at java.lang.Thread.run(Thread.java:748) {noformat} {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} {quote} Please also note that this errors was observed *right after* other error described in [WFLY-10758|https://issues.jboss.org/browse/WFLY-10758] so they may be related. Please refer to that issue for the sequence of events that led to the error. {quote} -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (DROOLS-2803) kie-wb-distribution: Investigate failing Unit Tests

by Toni Rikkola (JIRA)

[ https://issues.jboss.org/browse/DROOLS-2803?page=com.atlassian.jira.plugi... ] Toni Rikkola updated DROOLS-2803: --------------------------------- Sprint: 2018 Week 30-32 > kie-wb-distribution: Investigate failing Unit Tests > --------------------------------------------------- > > Key: DROOLS-2803 > URL: https://issues.jboss.org/browse/DROOLS-2803 > Project: Drools > Issue Type: Task > Components: integration > Affects Versions: 7.9.0.Final > Reporter: Michael Anstis > Assignee: Toni Rikkola > > See https://kie-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/kie-wb-distributi... > There are some repeat failing REST and Selenium Unit Tests failing. The _common consensus_ was to {{@Ignore}} tests and create a JIRA to track resolution. There are some PRs relating to the tests in progress already. > https://github.com/kiegroup/kie-wb-distributions/pull/782 > https://github.com/kiegroup/kie-wb-distributions/pull/778 -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10754) NullPointerException using Stateless with configured interceptors

by Matej Novotny (JIRA)

[ https://issues.jboss.org/browse/WFLY-10754?page=com.atlassian.jira.plugin... ] Matej Novotny commented on WFLY-10754: -------------------------------------- I have it the same way, I just copied the test over to [Weld TS|https://github.com/manovotn/core/tree/wfly10754] to run it there. WFLY 12 crashes for me in both setups - with or without EE 7 mode. What seems to work is WFLY 11 but that's hardly comparable as that one contains weld 2.4 whereas all the newer ones have Weld 3.x. But what I see is that if I remove {{OKInterceptor}} from {{beans.xml}} (it doesn't need to be there), then it works for me. Could you try that? > NullPointerException using Stateless with configured interceptors > ----------------------------------------------------------------- > > Key: WFLY-10754 > URL: https://issues.jboss.org/browse/WFLY-10754 > Project: WildFly > Issue Type: Bug > Components: CDI / Weld > Affects Versions: 13.0.0.Final > Environment: WildFly 13.0.0.Final and java 10.0.1 > Reporter: Luca Stancapiano > Assignee: Matej Novotny > > I report a strange behavior on WildFly 13 when configuring interceptors within stateless. Below I describe the scenario: > Here a simple interceptor: > {code:java} > package it.vige.injection.interceptors; > import javax.interceptor.AroundInvoke; > import javax.interceptor.Interceptor; > import javax.interceptor.InvocationContext; > @Interceptor > public class OKInterceptor { > @AroundInvoke > public Object aroundInvoke(InvocationContext ic) throws Exception { > return ic.proceed(); > } > } > {code} > Here an annotation used as interceptor binding: > {code:java} > package it.vige.injection.interceptors; > import static java.lang.annotation.ElementType.CONSTRUCTOR; > import static java.lang.annotation.ElementType.METHOD; > import static java.lang.annotation.ElementType.TYPE; > import static java.lang.annotation.RetentionPolicy.RUNTIME; > import java.lang.annotation.Retention; > import java.lang.annotation.Target; > import javax.interceptor.InterceptorBinding; > @Retention(RUNTIME) > @Target({ METHOD, TYPE, CONSTRUCTOR }) > @InterceptorBinding > public @interface NotOK { > } > {code} > Here an interceptor annotated with the interceptor binding: > {code:java} > package it.vige.injection.interceptors; > import javax.interceptor.AroundInvoke; > import javax.interceptor.Interceptor; > import javax.interceptor.InvocationContext; > @Interceptor > @NotOK > public class NotOKInterceptor { > @AroundInvoke > public Object aroundInvoke(InvocationContext ic) throws Exception { > return ic.proceed(); > } > } > {code} > Here the stateless service configured with both the interceptors: > {code:java} > package it.vige.injection.interceptors; > import javax.ejb.Stateless; > import javax.interceptor.Interceptors; > @Stateless > public class SimpleService { > @Interceptors({ OKInterceptor.class }) > public void ok() { > } > @NotOK > public void notOk() { > } > } > {code} > This service must have two methods, one attached to the simple interceptor and the other attached to the interceptor binding. > Here the beans.xml configuration: > {code:java} > <beans xmlns="http://xmlns.jcp.org/xml/ns/javaee" > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee > http://xmlns.jcp.org/xml/ns/javaee/beans_2_0.xsd" > version="2.0" bean-discovery-mode="all"> > <interceptors> > <class>it.vige.injection.interceptors.OKInterceptor</class> > <class>it.vige.injection.interceptors.NotOKInterceptor</class> > </interceptors> > </beans> > {code} > And in the end the client who call the service: > {code:java} > .... > @Inject > private SimpleService simpleService; > ... > // this call works: > simpleService.ok(); > // this call starts a NullPointerException: > simpleService.notOk(); > ... > {code} > when I try to call the notOk method I get this exception: > {code:java} > javax.ejb.EJBException: java.lang.NullPointerException > at deployment.test.war//it.vige.injection.test.InterceptorsTestCase.testNotOk(InterceptorsTestCase.java:52) > Caused by: java.lang.NullPointerException > at deployment.test.war//it.vige.injection.test.InterceptorsTestCase.testNotOk(InterceptorsTestCase.java:52) > {code} > The same thing was tested on WildFly 12.0.0.Final and it was ok. > If on WildFfly 13.0.0.Final I remove the @Stateless annotation from the service it works -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (DROOLS-2803) kie-wb-distribution: Investigate failing Unit Tests

by Michael Anstis (JIRA)

Michael Anstis created DROOLS-2803: -------------------------------------- Summary: kie-wb-distribution: Investigate failing Unit Tests Key: DROOLS-2803 URL: https://issues.jboss.org/browse/DROOLS-2803 Project: Drools Issue Type: Task Components: integration Affects Versions: 7.9.0.Final Reporter: Michael Anstis Assignee: Toni Rikkola See https://kie-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/kie-wb-distributi... There are some repeat failing REST and Selenium Unit Tests failing. The _common consensus_ was to {{@Ignore}} tests and create a JIRA to track resolution. There are some PRs relating to the tests in progress already. https://github.com/kiegroup/kie-wb-distributions/pull/782 https://github.com/kiegroup/kie-wb-distributions/pull/778 -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10761) Add <distributable/> to shared-session-config schema

by Paul Ferraro (JIRA)

Paul Ferraro created WFLY-10761: ----------------------------------- Summary: Add <distributable/> to shared-session-config schema Key: WFLY-10761 URL: https://issues.jboss.org/browse/WFLY-10761 Project: WildFly Issue Type: Feature Request Components: Clustering, Web (Undertow) Affects Versions: 10.0.0.CR2 Reporter: Paul Ferraro Assignee: Paul Ferraro Currently, the shared-session-config does not indicate whether a distributable or non-distributable session manager should be used. Consequently, we assign the session manager implementation based on the availability of the module providing the distributed implementation. Since the default infinispan web session configuration attempts to passivate sessions on shutdown, users will see NotSerializableExceptions on shutdown if they use non-serializable session attributes - even if their web applications do not define themselves as being <distributable/>. To clarify this ambiguity, we should add a <distributable/> element (a la web.xml) to the shared-session-config schema to determine which session manager implementation to use. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10759) UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10759?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10759: ----------------------------------- Description: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. The error was observed on node [dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...] : {noformat} [JBossINF] [0m[31m04:10:53,236 ERROR [io.undertow.servlet.request] (default task-203) UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException [JBossINF] at javax.faces.context.FacesContext.getExceptionHandler(FacesContext.java:304) [JBossINF] at javax.faces.event.ExceptionQueuedEventContext.getListenersForEventClass(ExceptionQueuedEventContext.java:272) [JBossINF] at com.sun.faces.application.ApplicationImpl.invokeComponentListenersFor(ApplicationImpl.java:2230) [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:337) [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:295) [JBossINF] at javax.faces.application.ApplicationWrapper.publishEvent(ApplicationWrapper.java:748) [JBossINF] at com.sun.faces.application.WebappLifecycleListener.requestDestroyed(WebappLifecycleListener.java:175) [JBossINF] at com.sun.faces.config.ConfigureListener.requestDestroyed(ConfigureListener.java:408) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestDestroyed(ApplicationListeners.java:260) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:335) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[33m04:10:53,236 WARN [org.jboss.weld.Servlet] (default task-203) WELD-000717: Unable to deactivate context org.jboss.weld.module.web.context.http.HttpRequestContextImpl@4a4b8c92 when destroying request HttpServletRequestImpl [ GET /clusterbench/ejbservlet ] [JBossINF] [0m[33m04:10:53,239 WARN [org.jboss.weld.Servlet] (default task-203) WELD-000717: Unable to deactivate context org.jboss.weld.module.web.context.http.HttpSessionContextImpl@1135e4b6 when destroying request HttpServletRequestImpl [ GET /clusterbench/ejbservlet ] [JBossINF] [0m[31m04:10:53,239 ERROR [io.undertow.request] (default task-203) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/ejbservlet request {Cookie=[$Version=0; JSESSIONID=3Nq88HEwcXMUhKudBoFAxcbXXnO6MZB7LCvXTBYZ.dev213; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[dev220:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Mon, 30 Jul 2018 08:10:32 GMT]}}: org.infinispan.commons.CacheException: javax.transaction.SystemException: Unable to rollback transaction [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:113) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:202) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: javax.transaction.SystemException: Unable to rollback transaction [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:135) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:108) [JBossINF] ... 27 more [JBossINF] Caused by: javax.transaction.HeuristicMixedException [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:457) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [JBossINF] ... 28 more [JBossINF] Caused by: javax.transaction.xa.XAException [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [JBossINF] ... 31 more [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:229) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:168) [JBossINF] ... 34 more [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336) [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] ... 1 more [JBossINF] Suppressed: org.infinispan.util.logging.TraceException [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:229) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:168) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:108) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:202) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] ... 1 more {noformat} {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} {quote} Please also note that this errors was observed after other errors like the one described in [WFLY-10758|https://issues.jboss.org/browse/WFLY-10758] so they may be related. {quote} was: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. The error was observed on node [dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...] : {noformat} [JBossINF] [0m[31m04:10:53,236 ERROR [io.undertow.servlet.request] (default task-203) UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException [JBossINF] at javax.faces.context.FacesContext.getExceptionHandler(FacesContext.java:304) [JBossINF] at javax.faces.event.ExceptionQueuedEventContext.getListenersForEventClass(ExceptionQueuedEventContext.java:272) [JBossINF] at com.sun.faces.application.ApplicationImpl.invokeComponentListenersFor(ApplicationImpl.java:2230) [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:337) [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:295) [JBossINF] at javax.faces.application.ApplicationWrapper.publishEvent(ApplicationWrapper.java:748) [JBossINF] at com.sun.faces.application.WebappLifecycleListener.requestDestroyed(WebappLifecycleListener.java:175) [JBossINF] at com.sun.faces.config.ConfigureListener.requestDestroyed(ConfigureListener.java:408) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestDestroyed(ApplicationListeners.java:260) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:335) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[33m04:10:53,236 WARN [org.jboss.weld.Servlet] (default task-203) WELD-000717: Unable to deactivate context org.jboss.weld.module.web.context.http.HttpRequestContextImpl@4a4b8c92 when destroying request HttpServletRequestImpl [ GET /clusterbench/ejbservlet ] [JBossINF] [0m[33m04:10:53,239 WARN [org.jboss.weld.Servlet] (default task-203) WELD-000717: Unable to deactivate context org.jboss.weld.module.web.context.http.HttpSessionContextImpl@1135e4b6 when destroying request HttpServletRequestImpl [ GET /clusterbench/ejbservlet ] [JBossINF] [0m[31m04:10:53,239 ERROR [io.undertow.request] (default task-203) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/ejbservlet request {Cookie=[$Version=0; JSESSIONID=3Nq88HEwcXMUhKudBoFAxcbXXnO6MZB7LCvXTBYZ.dev213; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[dev220:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Mon, 30 Jul 2018 08:10:32 GMT]}}: org.infinispan.commons.CacheException: javax.transaction.SystemException: Unable to rollback transaction [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:113) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:202) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: javax.transaction.SystemException: Unable to rollback transaction [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:135) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:108) [JBossINF] ... 27 more [JBossINF] Caused by: javax.transaction.HeuristicMixedException [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:457) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [JBossINF] ... 28 more [JBossINF] Caused by: javax.transaction.xa.XAException [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [JBossINF] ... 31 more [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:229) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:168) [JBossINF] ... 34 more [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336) [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] ... 1 more [JBossINF] Suppressed: org.infinispan.util.logging.TraceException [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:229) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:168) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:108) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:202) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] ... 1 more {noformat} {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} {quote} Please also note that this errors was observed after other errors like the one described [here|https://issues.jboss.org/browse/WFLY-10758] so may be related. {quote} > UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException > -------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: WFLY-10759 > URL: https://issues.jboss.org/browse/WFLY-10759 > Project: WildFly > Issue Type: Bug > Components: Clustering, Web (Undertow) > Affects Versions: 14.0.0.CR1 > Reporter: tommaso borgato > Assignee: Paul Ferraro > > The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. > The error was observed on node [dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...] : > {noformat} > [JBossINF] [0m[31m04:10:53,236 ERROR [io.undertow.servlet.request] (default task-203) UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException > [JBossINF] at javax.faces.context.FacesContext.getExceptionHandler(FacesContext.java:304) > [JBossINF] at javax.faces.event.ExceptionQueuedEventContext.getListenersForEventClass(ExceptionQueuedEventContext.java:272) > [JBossINF] at com.sun.faces.application.ApplicationImpl.invokeComponentListenersFor(ApplicationImpl.java:2230) > [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:337) > [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:295) > [JBossINF] at javax.faces.application.ApplicationWrapper.publishEvent(ApplicationWrapper.java:748) > [JBossINF] at com.sun.faces.application.WebappLifecycleListener.requestDestroyed(WebappLifecycleListener.java:175) > [JBossINF] at com.sun.faces.config.ConfigureListener.requestDestroyed(ConfigureListener.java:408) > [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestDestroyed(ApplicationListeners.java:260) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:335) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] > [JBossINF] [0m[33m04:10:53,236 WARN [org.jboss.weld.Servlet] (default task-203) WELD-000717: Unable to deactivate context org.jboss.weld.module.web.context.http.HttpRequestContextImpl@4a4b8c92 when destroying request HttpServletRequestImpl [ GET /clusterbench/ejbservlet ] > [JBossINF] [0m[33m04:10:53,239 WARN [org.jboss.weld.Servlet] (default task-203) WELD-000717: Unable to deactivate context org.jboss.weld.module.web.context.http.HttpSessionContextImpl@1135e4b6 when destroying request HttpServletRequestImpl [ GET /clusterbench/ejbservlet ] > [JBossINF] [0m[31m04:10:53,239 ERROR [io.undertow.request] (default task-203) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/ejbservlet request {Cookie=[$Version=0; JSESSIONID=3Nq88HEwcXMUhKudBoFAxcbXXnO6MZB7LCvXTBYZ.dev213; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[dev220:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Mon, 30 Jul 2018 08:10:32 GMT]}}: org.infinispan.commons.CacheException: javax.transaction.SystemException: Unable to rollback transaction > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:113) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:202) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) > [JBossINF] at java.lang.Thread.run(Thread.java:748) > [JBossINF] Caused by: javax.transaction.SystemException: Unable to rollback transaction > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:135) > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:108) > [JBossINF] ... 27 more > [JBossINF] Caused by: javax.transaction.HeuristicMixedException > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:457) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) > [JBossINF] ... 28 more > [JBossINF] Caused by: javax.transaction.xa.XAException > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) > [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) > [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) > [JBossINF] ... 31 more > [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 > [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) > [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:229) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:168) > [JBossINF] ... 34 more > [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336) > [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [JBossINF] ... 1 more > [JBossINF] Suppressed: org.infinispan.util.logging.TraceException > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) > [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) > [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:229) > [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:168) > [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) > [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) > [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) > [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:108) > [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:202) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) > [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) > [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) > [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) > [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) > [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) > [JBossINF] ... 1 more > {noformat} > {quote} > Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. > {quote} > {quote} > Please also note that this errors was observed after other errors like the one described in [WFLY-10758|https://issues.jboss.org/browse/WFLY-10758] so they may be related. > {quote} -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10758: ----------------------------------- Description: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * node dev212 left the cluster: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed: {noformat} [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} {quote} Please also note that when node dev213 left the cluster, node dev212 had not re-joined the cluster yet, or so it looks in the logs. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} was: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * node dev212 left the cluster: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed: {noformat} [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} > ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > ----------------------------------------------------------------------------------------------------- > > Key: WFLY-10758 > URL: https://issues.jboss.org/browse/WFLY-10758 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: tommaso borgato > Assignee: Paul Ferraro > > The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. > h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: > * the server started and the 4 nodes formed the cluster: > {noformat} > [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) > [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > {noformat} > * node dev212 left the cluster: > {noformat} > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > {noformat} > * rebalance with remaining nodes was concluded: > {noformat} > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > {noformat} > And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed: > {noformat} > [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > the block that is repeated over and over in the stack is the following: > {noformat} > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > {noformat} > 3396 sampling errors occurred in this run. > {quote} > Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. > {quote} > {quote} > Please also note that when node dev213 left the cluster, node dev212 had not re-joined the cluster yet, or so it looks in the logs. > {quote} > h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > This run was performed with the following jgroups parameters: > {noformat} > <protocol type="FD_ALL"> > <property name="timeout">5000</property> > <property name="interval">1000</property> > <property name="timeout_check_interval">2000</property> > </protocol> > <protocol type="VERIFY_SUSPECT"> > <property name="timeout">5000</property> > </protocol> > {noformat} > but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: > {noformat} > [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > unlike in the previous run, the stack block that is repeated over and over is: > {noformat} > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > {noformat} > -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10758: ----------------------------------- Description: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * node dev212 left the cluster: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed: {noformat} [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} was: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * node dev212 left the cluster: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} > ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > ----------------------------------------------------------------------------------------------------- > > Key: WFLY-10758 > URL: https://issues.jboss.org/browse/WFLY-10758 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: tommaso borgato > Assignee: Paul Ferraro > > The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. > h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: > * the server started and the 4 nodes formed the cluster: > {noformat} > [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) > [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > {noformat} > * node dev212 left the cluster: > {noformat} > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > {noformat} > * rebalance with remaining nodes was concluded: > {noformat} > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > {noformat} > And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed: > {noformat} > [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > the block that is repeated over and over in the stack is the following: > {noformat} > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > {noformat} > 3396 sampling errors occurred in this run. > {quote} > Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. > {quote} > h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > This run was performed with the following jgroups parameters: > {noformat} > <protocol type="FD_ALL"> > <property name="timeout">5000</property> > <property name="interval">1000</property> > <property name="timeout_check_interval">2000</property> > </protocol> > <protocol type="VERIFY_SUSPECT"> > <property name="timeout">5000</property> > </protocol> > {noformat} > but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: > {noformat} > [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > unlike in the previous run, the stack block that is repeated over and over is: > {noformat} > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > {noformat} > -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10758: ----------------------------------- Description: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * node dev212 left the cluster: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} was: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} > ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > ----------------------------------------------------------------------------------------------------- > > Key: WFLY-10758 > URL: https://issues.jboss.org/browse/WFLY-10758 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: tommaso borgato > Assignee: Paul Ferraro > > The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. > h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: > * the server started and the 4 nodes formed the cluster: > {noformat} > [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) > [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > {noformat} > * node dev212 left the cluster: > {noformat} > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > {noformat} > * rebalance with remaining nodes was concluded: > {noformat} > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > {noformat} > And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: > {noformat} > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service > [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service > [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > the block that is repeated over and over in the stack is the following: > {noformat} > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > {noformat} > 3396 sampling errors occurred in this run. > {quote} > Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. > {quote} > h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > This run was performed with the following jgroups parameters: > {noformat} > <protocol type="FD_ALL"> > <property name="timeout">5000</property> > <property name="interval">1000</property> > <property name="timeout_check_interval">2000</property> > </protocol> > <protocol type="VERIFY_SUSPECT"> > <property name="timeout">5000</property> > </protocol> > {noformat} > but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: > {noformat} > [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > unlike in the previous run, the stack block that is repeated over and over is: > {noformat} > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > {noformat} > -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

jboss-jira July 2018