[JBoss JIRA] (WFLY-10754) NullPointerException using Stateless with configured interceptors
by Luca Stancapiano (JIRA)
[ https://issues.jboss.org/browse/WFLY-10754?page=com.atlassian.jira.plugin... ]
Luca Stancapiano commented on WFLY-10754:
-----------------------------------------
hi Matej.... I start WildFly through the basic Arquillian configuration as you can see in the online test. Maybe something wrong in the Arquillian configuration. Can you try to start the test as described in the Steps to Reproduce of the issue?
> NullPointerException using Stateless with configured interceptors
> -----------------------------------------------------------------
>
> Key: WFLY-10754
> URL: https://issues.jboss.org/browse/WFLY-10754
> Project: WildFly
> Issue Type: Bug
> Components: CDI / Weld
> Affects Versions: 13.0.0.Final
> Environment: WildFly 13.0.0.Final and java 10.0.1
> Reporter: Luca Stancapiano
> Assignee: Matej Novotny
>
> I report a strange behavior on WildFly 13 when configuring interceptors within stateless. Below I describe the scenario:
> Here a simple interceptor:
> {code:java}
> package it.vige.injection.interceptors;
> import javax.interceptor.AroundInvoke;
> import javax.interceptor.Interceptor;
> import javax.interceptor.InvocationContext;
> @Interceptor
> public class OKInterceptor {
> @AroundInvoke
> public Object aroundInvoke(InvocationContext ic) throws Exception {
> return ic.proceed();
> }
> }
> {code}
> Here an annotation used as interceptor binding:
> {code:java}
> package it.vige.injection.interceptors;
> import static java.lang.annotation.ElementType.CONSTRUCTOR;
> import static java.lang.annotation.ElementType.METHOD;
> import static java.lang.annotation.ElementType.TYPE;
> import static java.lang.annotation.RetentionPolicy.RUNTIME;
> import java.lang.annotation.Retention;
> import java.lang.annotation.Target;
> import javax.interceptor.InterceptorBinding;
> @Retention(RUNTIME)
> @Target({ METHOD, TYPE, CONSTRUCTOR })
> @InterceptorBinding
> public @interface NotOK {
> }
> {code}
> Here an interceptor annotated with the interceptor binding:
> {code:java}
> package it.vige.injection.interceptors;
> import javax.interceptor.AroundInvoke;
> import javax.interceptor.Interceptor;
> import javax.interceptor.InvocationContext;
> @Interceptor
> @NotOK
> public class NotOKInterceptor {
> @AroundInvoke
> public Object aroundInvoke(InvocationContext ic) throws Exception {
> return ic.proceed();
> }
> }
> {code}
> Here the stateless service configured with both the interceptors:
> {code:java}
> package it.vige.injection.interceptors;
> import javax.ejb.Stateless;
> import javax.interceptor.Interceptors;
> @Stateless
> public class SimpleService {
> @Interceptors({ OKInterceptor.class })
> public void ok() {
> }
> @NotOK
> public void notOk() {
> }
> }
> {code}
> This service must have two methods, one attached to the simple interceptor and the other attached to the interceptor binding.
> Here the beans.xml configuration:
> {code:java}
> <beans xmlns="http://xmlns.jcp.org/xml/ns/javaee"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee
> http://xmlns.jcp.org/xml/ns/javaee/beans_2_0.xsd"
> version="2.0" bean-discovery-mode="all">
> <interceptors>
> <class>it.vige.injection.interceptors.OKInterceptor</class>
> <class>it.vige.injection.interceptors.NotOKInterceptor</class>
> </interceptors>
> </beans>
> {code}
> And in the end the client who call the service:
> {code:java}
> ....
> @Inject
> private SimpleService simpleService;
> ...
> // this call works:
> simpleService.ok();
> // this call starts a NullPointerException:
> simpleService.notOk();
> ...
> {code}
> when I try to call the notOk method I get this exception:
> {code:java}
> javax.ejb.EJBException: java.lang.NullPointerException
> at deployment.test.war//it.vige.injection.test.InterceptorsTestCase.testNotOk(InterceptorsTestCase.java:52)
> Caused by: java.lang.NullPointerException
> at deployment.test.war//it.vige.injection.test.InterceptorsTestCase.testNotOk(InterceptorsTestCase.java:52)
> {code}
> The same thing was tested on WildFly 12.0.0.Final and it was ok.
> If on WildFfly 13.0.0.Final I remove the @Stateless annotation from the service it works
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (DROOLS-1663) Kie DMN doesn't support IMPORT decisions between DMN files
by Tibor Zimányi (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1663?page=com.atlassian.jira.plugi... ]
Tibor Zimányi updated DROOLS-1663:
----------------------------------
Tester: Tibor Zimányi
> Kie DMN doesn't support IMPORT decisions between DMN files
> ----------------------------------------------------------
>
> Key: DROOLS-1663
> URL: https://issues.jboss.org/browse/DROOLS-1663
> Project: Drools
> Issue Type: Enhancement
> Components: dmn engine
> Reporter: Stylianos Koussouris
> Assignee: Matteo Mortari
> Attachments: IMG_2197.jpg, IMG_2198.jpg, IMG_2199.jpg, dmn import defaults.odt
>
>
> DMN Spec 1.1
> Page 40.
> import: Import [*] This attribute is used to import externally defined elements and
> make them available for use by elements in this Definitions.
> Section 6.3.3 Import metamodel
> The aim here is to be able to import one Decision defined in a separate DMN into another where it is used as a supporting decision and is referenced (RequiredDecision)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (DROOLS-1701) FEEL code generation and compilation
by Tibor Zimányi (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1701?page=com.atlassian.jira.plugi... ]
Tibor Zimányi updated DROOLS-1701:
----------------------------------
Tester: Tibor Zimányi
> FEEL code generation and compilation
> ------------------------------------
>
> Key: DROOLS-1701
> URL: https://issues.jboss.org/browse/DROOLS-1701
> Project: Drools
> Issue Type: Task
> Components: dmn engine
> Reporter: Matteo Mortari
> Assignee: Matteo Mortari
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (WFLY-10754) NullPointerException using Stateless with configured interceptors
by Matej Novotny (JIRA)
[ https://issues.jboss.org/browse/WFLY-10754?page=com.atlassian.jira.plugin... ]
Matej Novotny commented on WFLY-10754:
--------------------------------------
To me it seems even WildFly 12 is affected, at least the pristine Final version.
[~sviluppatorefico] Did you have somehow customized WFLY?
> NullPointerException using Stateless with configured interceptors
> -----------------------------------------------------------------
>
> Key: WFLY-10754
> URL: https://issues.jboss.org/browse/WFLY-10754
> Project: WildFly
> Issue Type: Bug
> Components: CDI / Weld
> Affects Versions: 13.0.0.Final
> Environment: WildFly 13.0.0.Final and java 10.0.1
> Reporter: Luca Stancapiano
> Assignee: Matej Novotny
>
> I report a strange behavior on WildFly 13 when configuring interceptors within stateless. Below I describe the scenario:
> Here a simple interceptor:
> {code:java}
> package it.vige.injection.interceptors;
> import javax.interceptor.AroundInvoke;
> import javax.interceptor.Interceptor;
> import javax.interceptor.InvocationContext;
> @Interceptor
> public class OKInterceptor {
> @AroundInvoke
> public Object aroundInvoke(InvocationContext ic) throws Exception {
> return ic.proceed();
> }
> }
> {code}
> Here an annotation used as interceptor binding:
> {code:java}
> package it.vige.injection.interceptors;
> import static java.lang.annotation.ElementType.CONSTRUCTOR;
> import static java.lang.annotation.ElementType.METHOD;
> import static java.lang.annotation.ElementType.TYPE;
> import static java.lang.annotation.RetentionPolicy.RUNTIME;
> import java.lang.annotation.Retention;
> import java.lang.annotation.Target;
> import javax.interceptor.InterceptorBinding;
> @Retention(RUNTIME)
> @Target({ METHOD, TYPE, CONSTRUCTOR })
> @InterceptorBinding
> public @interface NotOK {
> }
> {code}
> Here an interceptor annotated with the interceptor binding:
> {code:java}
> package it.vige.injection.interceptors;
> import javax.interceptor.AroundInvoke;
> import javax.interceptor.Interceptor;
> import javax.interceptor.InvocationContext;
> @Interceptor
> @NotOK
> public class NotOKInterceptor {
> @AroundInvoke
> public Object aroundInvoke(InvocationContext ic) throws Exception {
> return ic.proceed();
> }
> }
> {code}
> Here the stateless service configured with both the interceptors:
> {code:java}
> package it.vige.injection.interceptors;
> import javax.ejb.Stateless;
> import javax.interceptor.Interceptors;
> @Stateless
> public class SimpleService {
> @Interceptors({ OKInterceptor.class })
> public void ok() {
> }
> @NotOK
> public void notOk() {
> }
> }
> {code}
> This service must have two methods, one attached to the simple interceptor and the other attached to the interceptor binding.
> Here the beans.xml configuration:
> {code:java}
> <beans xmlns="http://xmlns.jcp.org/xml/ns/javaee"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee
> http://xmlns.jcp.org/xml/ns/javaee/beans_2_0.xsd"
> version="2.0" bean-discovery-mode="all">
> <interceptors>
> <class>it.vige.injection.interceptors.OKInterceptor</class>
> <class>it.vige.injection.interceptors.NotOKInterceptor</class>
> </interceptors>
> </beans>
> {code}
> And in the end the client who call the service:
> {code:java}
> ....
> @Inject
> private SimpleService simpleService;
> ...
> // this call works:
> simpleService.ok();
> // this call starts a NullPointerException:
> simpleService.notOk();
> ...
> {code}
> when I try to call the notOk method I get this exception:
> {code:java}
> javax.ejb.EJBException: java.lang.NullPointerException
> at deployment.test.war//it.vige.injection.test.InterceptorsTestCase.testNotOk(InterceptorsTestCase.java:52)
> Caused by: java.lang.NullPointerException
> at deployment.test.war//it.vige.injection.test.InterceptorsTestCase.testNotOk(InterceptorsTestCase.java:52)
> {code}
> The same thing was tested on WildFly 12.0.0.Final and it was ok.
> If on WildFfly 13.0.0.Final I remove the @Stateless annotation from the service it works
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ]
tommaso borgato updated WFLY-10758:
-----------------------------------
Description:
The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the server was started and the first node ({{*dev212*}}) was failed:
{noformat}
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
[JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
[JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
the block that is repeated over and over in the stack is the following:
{noformat}
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
{noformat}
3396 sampling errors occurred in this run.
{quote}
Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
{quote}
h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
This run was performed with the following jgroups parameters:
{noformat}
<protocol type="FD_ALL">
<property name="timeout">5000</property>
<property name="interval">1000</property>
<property name="timeout_check_interval">2000</property>
</protocol>
<protocol type="VERIFY_SUSPECT">
<property name="timeout">5000</property>
</protocol>
{noformat}
but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]:
{noformat}
[JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151)
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
unlike in the previous run, the stack block that is repeated over and over is:
{noformat}
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
{noformat}
was:
The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the server was started and the first node ({{*dev212*}}) was failed:
{noformat}
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
[JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
[JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
the block that is repeated over and over in the stack is the following:
{noformat}
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
{noformat}
3396 sampling errors occurred in this run.
Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
This run was performed with the following jgroups parameters:
{noformat}
<protocol type="FD_ALL">
<property name="timeout">5000</property>
<property name="interval">1000</property>
<property name="timeout_check_interval">2000</property>
</protocol>
<protocol type="VERIFY_SUSPECT">
<property name="timeout">5000</property>
</protocol>
{noformat}
but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]:
{noformat}
[JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151)
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
unlike in the previous run, the stack block that is repeated over and over is:
{noformat}
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
{noformat}
> ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> -----------------------------------------------------------------------------------------------------
>
> Key: WFLY-10758
> URL: https://issues.jboss.org/browse/WFLY-10758
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: tommaso borgato
> Assignee: Paul Ferraro
>
> The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
> h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
> The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the server was started and the first node ({{*dev212*}}) was failed:
> {noformat}
> [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
> [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
> [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
> [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> ...
> {noformat}
> the block that is repeated over and over in the stack is the following:
> {noformat}
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> {noformat}
> 3396 sampling errors occurred in this run.
> {quote}
> Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
> {quote}
> h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
> This run was performed with the following jgroups parameters:
> {noformat}
> <protocol type="FD_ALL">
> <property name="timeout">5000</property>
> <property name="interval">1000</property>
> <property name="timeout_check_interval">2000</property>
> </protocol>
> <protocol type="VERIFY_SUSPECT">
> <property name="timeout">5000</property>
> </protocol>
> {noformat}
> but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]:
> {noformat}
> [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151)
> [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
> [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> ...
> {noformat}
> unlike in the previous run, the stack block that is repeated over and over is:
> {noformat}
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ]
tommaso borgato updated WFLY-10758:
-----------------------------------
Description:
The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the server was started and the first node ({{*dev212*}}) was failed:
{noformat}
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
[JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
[JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
the block that is repeated over and over in the stack is the following:
{noformat}
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
{noformat}
3396 sampling errors occurred in this run.
Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
This run was performed with the following jgroups parameters:
{noformat}
<protocol type="FD_ALL">
<property name="timeout">5000</property>
<property name="interval">1000</property>
<property name="timeout_check_interval">2000</property>
</protocol>
<protocol type="VERIFY_SUSPECT">
<property name="timeout">5000</property>
</protocol>
{noformat}
but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]:
{noformat}
[JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151)
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
unlike in the previous run, the stack block that is repeated over and over is:
{noformat}
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
{noformat}
was:
The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the server was started and the first node ({{*dev212*}}) was failed:
{noformat}
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
[JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
[JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
[JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
[JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
[JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
[JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
[JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
[JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
the repeated block in the stack is the following:
{noformat}
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
{noformat}
3396 sampling errors occurred in this run.
Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
> ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> -----------------------------------------------------------------------------------------------------
>
> Key: WFLY-10758
> URL: https://issues.jboss.org/browse/WFLY-10758
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: tommaso borgato
> Assignee: Paul Ferraro
>
> The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
> h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
> The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the server was started and the first node ({{*dev212*}}) was failed:
> {noformat}
> [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14
> [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15
> [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16
> [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
> [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service
> [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17
> [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18
> [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
> [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> ...
> {noformat}
> the block that is repeated over and over in the stack is the following:
> {noformat}
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> {noformat}
> 3396 sampling errors occurred in this run.
> Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
> h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
> This run was performed with the following jgroups parameters:
> {noformat}
> <protocol type="FD_ALL">
> <property name="timeout">5000</property>
> <property name="interval">1000</property>
> <property name="timeout_check_interval">2000</property>
> </protocol>
> <protocol type="VERIFY_SUSPECT">
> <property name="timeout">5000</property>
> </protocol>
> {noformat}
> but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]:
> {noformat}
> [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151)
> [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
> [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> ...
> {noformat}
> unlike in the previous run, the stack block that is repeated over and over is:
> {noformat}
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
Paul Ferraro commented on WFLY-10736:
-------------------------------------
[~mnovak]
{quote}This use case does not seem to be so uncommon in production. In case that any liveness check of server fails there are scripts which simply restart the server. Yes, it can try clean shutdown first but if server does not shutdown due to any reason in given timeout then it will kill and start server again which is the same scenario.{quote}
In the same way that we recommend staggering the startup of cluster nodes to facilitate the cluster joining process, those scripts should also be modified to include a brief delay between kill and start.
{quote}
Also this issue is regression. Those tests with kill and restart scenario are executed since EAP 6 times and there were no issues until now. There is used default config for infinispan and JGroups.
{quote}
I acknowledge that this might be a regression - however, I'm trying to determine if the workaround is sufficient to lower the priority from blocker. State-transfer timeouts causing systems to fail to start have plagued customers in the past. We exposed the ability to configure state-transfer in a non-blocking fashion for that reason.
{quote}
I tried the workaround [...] and it seems to work. I can still see the exception on node-2 however node-1 does not hang any more.
{quote}
You will always see clustering related exceptions following a failure condition. However, I want to make sure that the system recovers adequately (i.e. beyond not handing on startup). Can you attach logs from node-1 & node-2 for this run?
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Attachments: clusterKilTest.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (DROOLS-2795) [DMN Designer] Data-types: Grid: DecisionTable
by Jozef Marko (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2795?page=com.atlassian.jira.plugi... ]
Jozef Marko updated DROOLS-2795:
--------------------------------
Description:
*_Decision Table_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
- (x) Hide Output Data Type in header when there are multiple {{OutputClause}} columns
- (x) {{InputClause}} columns header should show Input Data Type
- (x) {{InputClause}} columns should support changing the Input Data Type
- (x) {{OutputClause}} columns header should show Output Data Type
- (x) {{OutputClause}} columns should support changing the Output Data Type
was:
*_Decision Table_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
- (x) Hide Output Data Type in header when there are multiple {{OutputClause}} columns
- (x) {{InputClause}} columns header should show Input Data Type
- (x) {{InputClause}} columns should support changing the Input Data Type
- (x) {{OutputClause}} columns header should show Input Data Type
- (x) {{OutputClause}} columns should support changing the Input Data Type
> [DMN Designer] Data-types: Grid: DecisionTable
> ----------------------------------------------
>
> Key: DROOLS-2795
> URL: https://issues.jboss.org/browse/DROOLS-2795
> Project: Drools
> Issue Type: Feature Request
> Components: DMN Editor
> Reporter: Michael Anstis
> Assignee: Michael Anstis
>
> *_Decision Table_*
> - (x) Grid header _could_ show Output Data Type
> - (/) Editing Output Data Type is possible via Properties panel
> - (x) Hide Output Data Type in header when there are multiple {{OutputClause}} columns
> - (x) {{InputClause}} columns header should show Input Data Type
> - (x) {{InputClause}} columns should support changing the Input Data Type
> - (x) {{OutputClause}} columns header should show Output Data Type
> - (x) {{OutputClause}} columns should support changing the Output Data Type
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (DROOLS-2794) [DMN Designer] Data-types: Support usage in grids
by Michael Anstis (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2794?page=com.atlassian.jira.plugi... ]
Michael Anstis updated DROOLS-2794:
-----------------------------------
Description:
Listing requirements here; and then possibly move this to an EPIC and add smaller tasks.
*_General_*
- (x) {{<< Back to XXX}} does not update if name changed in Properties panel
- (x) "TypeRef" on Properties panel should show "Output Data Type"
*_Literal Expression_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
- (x) Update header when Output Data Type is changed via Properties panel
*_Decision Table_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
- (x) Hide Output Data Type in header when there are multiple {{OutputClause}} columns
- (x) {{InputClause}} columns header should show Input Data Type
- (x) {{InputClause}} columns should support changing the Input Data Type
- (x) {{OutputClause}} columns header should show Output Data Type
- (x) {{OutputClause}} columns should support changing the Output Data Type
*_Context_*
- (x) Grid header _could_ show Output Data Type (for Decision/BKM)
- (/) Editing Output Data Type (for Decision/BKM) is possible via Properties panel
- (x) {{ContextEntry}} should show Output Data Type (for {{ContextEntry}})
- (x) {{ContextEntry}} should support changing the Output Data Type (for {{ContextEntry}})
*_Relation_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
- (x) {{InformationItem}} column should show Input Data Type
- (x) {{InformationItem}} column should support changing Input Data Type
*_Invocation_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
was:
Listing requirements here; and then possibly move this to an EPIC and add smaller tasks.
*_General_*
- (x) {{<< Back to XXX}} does not update if name changed in Properties panel
- (x) "TypeRef" on Properties panel should show "Output Data Type"
*_Literal Expression_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
- (x) Update header when Output Data Type is changed via Properties panel
*_Decision Table_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
- (x) Hide Output Data Type in header when there are multiple {{OutputClause}} columns
- (x) {{InputClause}} columns header should show Input Data Type
- (x) {{InputClause}} columns should support changing the Input Data Type
- (x) {{OutputClause}} columns header should show Input Data Type
- (x) {{OutputClause}} columns should support changing the Input Data Type
*_Context_*
- (x) Grid header _could_ show Output Data Type (for Decision/BKM)
- (/) Editing Output Data Type (for Decision/BKM) is possible via Properties panel
- (x) {{ContextEntry}} should show Output Data Type (for {{ContextEntry}})
- (x) {{ContextEntry}} should support changing the Output Data Type (for {{ContextEntry}})
*_Relation_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
- (x) {{InformationItem}} column should show Input Data Type
- (x) {{InformationItem}} column should support changing Input Data Type
*_Invocation_*
- (x) Grid header _could_ show Output Data Type
- (/) Editing Output Data Type is possible via Properties panel
> [DMN Designer] Data-types: Support usage in grids
> -------------------------------------------------
>
> Key: DROOLS-2794
> URL: https://issues.jboss.org/browse/DROOLS-2794
> Project: Drools
> Issue Type: Epic
> Components: DMN Editor
> Affects Versions: 7.9.0.Final
> Reporter: Michael Anstis
> Assignee: Michael Anstis
>
> Listing requirements here; and then possibly move this to an EPIC and add smaller tasks.
> *_General_*
> - (x) {{<< Back to XXX}} does not update if name changed in Properties panel
> - (x) "TypeRef" on Properties panel should show "Output Data Type"
> *_Literal Expression_*
> - (x) Grid header _could_ show Output Data Type
> - (/) Editing Output Data Type is possible via Properties panel
> - (x) Update header when Output Data Type is changed via Properties panel
> *_Decision Table_*
> - (x) Grid header _could_ show Output Data Type
> - (/) Editing Output Data Type is possible via Properties panel
> - (x) Hide Output Data Type in header when there are multiple {{OutputClause}} columns
> - (x) {{InputClause}} columns header should show Input Data Type
> - (x) {{InputClause}} columns should support changing the Input Data Type
> - (x) {{OutputClause}} columns header should show Output Data Type
> - (x) {{OutputClause}} columns should support changing the Output Data Type
> *_Context_*
> - (x) Grid header _could_ show Output Data Type (for Decision/BKM)
> - (/) Editing Output Data Type (for Decision/BKM) is possible via Properties panel
> - (x) {{ContextEntry}} should show Output Data Type (for {{ContextEntry}})
> - (x) {{ContextEntry}} should support changing the Output Data Type (for {{ContextEntry}})
> *_Relation_*
> - (x) Grid header _could_ show Output Data Type
> - (/) Editing Output Data Type is possible via Properties panel
> - (x) {{InformationItem}} column should show Input Data Type
> - (x) {{InformationItem}} column should support changing Input Data Type
> *_Invocation_*
> - (x) Grid header _could_ show Output Data Type
> - (/) Editing Output Data Type is possible via Properties panel
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0
[JBoss JIRA] (DROOLS-2691) Data model persistence
by Daniele Zonca (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2691?page=com.atlassian.jira.plugi... ]
Daniele Zonca closed DROOLS-2691.
---------------------------------
Resolution: Duplicate Issue
> Data model persistence
> ----------------------
>
> Key: DROOLS-2691
> URL: https://issues.jboss.org/browse/DROOLS-2691
> Project: Drools
> Issue Type: Task
> Components: Scenario Simulation and Testing
> Reporter: Ivo Bek
> Assignee: Daniele Zonca
>
> Store data model of the scenario model to the asset ( marshaling )
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
- 1
- 0