jboss-jira July 2018

jboss-jira@lists.jboss.org

1 participants
2474 discussions

[JBoss JIRA] (WFLY-10760) Formalize web session clustering modules into a proper subsystem

by Paul Ferraro (JIRA)

Paul Ferraro created WFLY-10760: ----------------------------------- Summary: Formalize web session clustering modules into a proper subsystem Key: WFLY-10760 URL: https://issues.jboss.org/browse/WFLY-10760 Project: WildFly Issue Type: Enhancement Components: Clustering Affects Versions: 10.0.0.CR3 Reporter: Paul Ferraro Assignee: Paul Ferraro Currently, the coupling between the undertow subsystem and the modules required for distributed web session manager and single sign-on manager support is very loose. Consequently, misconfiguration (e.g. a missing "web" cache-container) can prevent deployment from succeeding without an adequate explanation. The subsystem would define the requisite cache-container, exposed as a capability. This subsystem would exposes a number of profiles, containing the configuration traditionally specified in jboss-web.xml, as well as the cache configuration to use (specified by cache-container + cache name). jboss-web.xml would only need to reference a profile by name, or, if unspecified, use the default profile. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10758: ----------------------------------- Description: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} was: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: # the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} # rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} > ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > ----------------------------------------------------------------------------------------------------- > > Key: WFLY-10758 > URL: https://issues.jboss.org/browse/WFLY-10758 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: tommaso borgato > Assignee: Paul Ferraro > > The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. > h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: > * the server started and the 4 nodes formed the cluster: > {noformat} > [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) > [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > {noformat} > * rebalance with remaining nodes was concluded: > {noformat} > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > {noformat} > And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: > {noformat} > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service > [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service > [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > the block that is repeated over and over in the stack is the following: > {noformat} > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > {noformat} > 3396 sampling errors occurred in this run. > {quote} > Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. > {quote} > h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > This run was performed with the following jgroups parameters: > {noformat} > <protocol type="FD_ALL"> > <property name="timeout">5000</property> > <property name="interval">1000</property> > <property name="timeout_check_interval">2000</property> > </protocol> > <protocol type="VERIFY_SUSPECT"> > <property name="timeout">5000</property> > </protocol> > {noformat} > but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: > {noformat} > [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > unlike in the previous run, the stack block that is repeated over and over is: > {noformat} > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > {noformat} > -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10758: ----------------------------------- Description: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: # the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} # rebalance with remaining nodes was concluded: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} was: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: the server started and the 4 nodes cluster was formed: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} node dev212 left the cluster: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 {noformat} rebalance with remaining nodes was concluded cluster: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} > ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > ----------------------------------------------------------------------------------------------------- > > Key: WFLY-10758 > URL: https://issues.jboss.org/browse/WFLY-10758 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: tommaso borgato > Assignee: Paul Ferraro > > The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. > h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: > # the server started and the 4 nodes formed the cluster: > {noformat} > [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) > [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > {noformat} > # rebalance with remaining nodes was concluded: > {noformat} > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > {noformat} > And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: > {noformat} > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service > [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service > [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > the block that is repeated over and over in the stack is the following: > {noformat} > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > {noformat} > 3396 sampling errors occurred in this run. > {quote} > Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. > {quote} > h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > This run was performed with the following jgroups parameters: > {noformat} > <protocol type="FD_ALL"> > <property name="timeout">5000</property> > <property name="interval">1000</property> > <property name="timeout_check_interval">2000</property> > </protocol> > <protocol type="VERIFY_SUSPECT"> > <property name="timeout">5000</property> > </protocol> > {noformat} > but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: > {noformat} > [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > unlike in the previous run, the stack block that is repeated over and over is: > {noformat} > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > {noformat} > -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError

by tommaso borgato (JIRA)

[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10758: ----------------------------------- Description: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: the server started and the 4 nodes cluster was formed: {noformat} [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} node dev212 left the cluster: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 {noformat} rebalance with remaining nodes was concluded cluster: {noformat} [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} was: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the server was started and the first node ({{*dev212*}}) was failed: {noformat} [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} > ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > ----------------------------------------------------------------------------------------------------- > > Key: WFLY-10758 > URL: https://issues.jboss.org/browse/WFLY-10758 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: tommaso borgato > Assignee: Paul Ferraro > > The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. > h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: > the server started and the 4 nodes cluster was formed: > {noformat} > [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) > [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] > [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] > [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster > {noformat} > node dev212 left the cluster: > {noformat} > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > {noformat} > rebalance with remaining nodes was concluded cluster: > {noformat} > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > {noformat} > And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the first node ({{*dev212*}}) was failed: > {noformat} > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] > [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m04:08:13,770 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,771 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,781 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,782 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,789 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,795 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t37) [Context=client-mappings] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,797 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,801 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,804 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,809 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t8) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,814 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t63) [Context=default-server] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,818 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t1) [Context=default] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,825 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t46) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,828 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t21) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,836 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t5) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,838 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:13,847 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,857 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,858 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:13,863 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t6) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,877 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t3) [Context=default] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:13,878 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t15) [Context=client-mappings] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,893 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t8) [Context=default-server] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,918 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t2) [Context=default] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:13,943 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [dev212, dev215, dev213, dev214] with topology id 14 > [JBossINF] [0m[0m04:08:13,944 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [dev215, dev213, dev214], topology id 15 > [JBossINF] [0m[0m04:08:13,947 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t62) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [dev215, dev213, dev214], phase READ_OLD_WRITE_ALL, topology id 16 > [JBossINF] [0m[0m04:08:14,004 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,069 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t29) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,103 INFO [org.infinispan.CLUSTER] (transport-thread--p13-t2) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,109 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t7) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,127 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t28) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,175 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t20) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,219 INFO [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0003: dev215 elected as the singleton provider of the org.wildfly.ejb3.clustered.singleton service > [JBossINF] [0m[0m04:08:14,237 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t17) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,260 INFO [org.wildfly.clustering.server] (ChannelCommandDispatcherFactory - 17) WFLYCLSV0001: This node will now operate as the singleton provider of the org.wildfly.ejb3.clustered.singleton service > [JBossINF] [0m[0m04:08:14,284 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t5) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,296 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t27) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 17 > [JBossINF] [0m[0m04:08:14,356 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t23) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 18 > [JBossINF] [0m[0m04:08:14,358 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t21) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 > [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] > [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster > [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > the block that is repeated over and over in the stack is the following: > {noformat} > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > {noformat} > 3396 sampling errors occurred in this run. > {quote} > Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. > {quote} > h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] > This run was performed with the following jgroups parameters: > {noformat} > <protocol type="FD_ALL"> > <property name="timeout">5000</property> > <property name="interval">1000</property> > <property name="timeout_check_interval">2000</property> > </protocol> > <protocol type="VERIFY_SUSPECT"> > <property name="timeout">5000</property> > </protocol> > {noformat} > but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: > {noformat} > [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] > [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster > [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) > [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) > [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) > [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) > [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) > [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) > [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) > [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) > [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) > [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) > [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) > [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) > [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) > [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > ... > {noformat} > unlike in the previous run, the stack block that is repeated over and over is: > {noformat} > [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) > [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) > [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) > [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) > {noformat} > -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill

by Miroslav Novak (JIRA)

[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ] Miroslav Novak updated WFLY-10736: ---------------------------------- Attachment: logs-with-workaround.zip > Server in cluster hangs during start after previous kill > -------------------------------------------------------- > > Key: WFLY-10736 > URL: https://issues.jboss.org/browse/WFLY-10736 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: Miroslav Novak > Assignee: Paul Ferraro > Priority: Blocker > Labels: blocker-WF14 > Attachments: clusterKilTest.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, standalone-full-ha-1.xml, standalone-full-ha-2.xml > > > There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run. > Test Scenario: > * There are two servers. InQueue is deployed on Node 1, > * OutQueue is deployed on Node 2. Both servers are started. > * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2. > * Node 1 is killed and started again. > * All messages are received from OutQueue deployed on Node 2. > Result: > Node 1 does not start after kill and hangs. There is following exception logged in node 2: > {code} > :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster > 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]} > 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17 > 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17 > 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17 > 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131] > Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131] > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131] > at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) > at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71) > at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540) > at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523) > at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334) > at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85) > at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745) > at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) > at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) > at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) > ... 1 more > Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > ... 1 more > [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1] > 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131] > Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131] > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131] > at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) > at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71) > at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540) > at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523) > at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334) > at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85) > at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745) > at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) > at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) > at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) > ... 1 more > Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > ... 1 more > [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1] > {code} > There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000. > Attaching thread dump from node 1 when it hangs during start. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill

by Miroslav Novak (JIRA)

[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ] Miroslav Novak updated WFLY-10736: ---------------------------------- Attachment: (was: logs-with-workaround.zip) > Server in cluster hangs during start after previous kill > -------------------------------------------------------- > > Key: WFLY-10736 > URL: https://issues.jboss.org/browse/WFLY-10736 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: Miroslav Novak > Assignee: Paul Ferraro > Priority: Blocker > Labels: blocker-WF14 > Attachments: clusterKilTest.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, standalone-full-ha-1.xml, standalone-full-ha-2.xml > > > There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run. > Test Scenario: > * There are two servers. InQueue is deployed on Node 1, > * OutQueue is deployed on Node 2. Both servers are started. > * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2. > * Node 1 is killed and started again. > * All messages are received from OutQueue deployed on Node 2. > Result: > Node 1 does not start after kill and hangs. There is following exception logged in node 2: > {code} > :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster > 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]} > 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17 > 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17 > 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17 > 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131] > Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131] > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131] > at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) > at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71) > at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540) > at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523) > at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334) > at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85) > at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745) > at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) > at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) > at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) > ... 1 more > Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > ... 1 more > [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1] > 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131] > Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131] > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131] > at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) > at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71) > at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540) > at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523) > at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334) > at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85) > at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745) > at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) > at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) > at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) > ... 1 more > Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > ... 1 more > [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1] > {code} > There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000. > Attaching thread dump from node 1 when it hangs during start. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill

by Miroslav Novak (JIRA)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill

by Miroslav Novak (JIRA)

[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ] Miroslav Novak commented on WFLY-10736: --------------------------------------- [~pferraro] I tried the workaround: /subsytem=infinispan/cache-container=server/replicated-cache=default/component=state-transfer:write-attribute(name=timeout, value=0) and it works. I cannot reproduce this issue again (node-1 does not hang during start). However It's unclear to me how it affect clustering caches. It seems that setting {{timeout}} to 0 just does not wait for sync of replication cache at start and server starts immediately. Will cache sync after start-up? {quote} Can you attach logs from node-1 & node-2 for this run? {quote} Attaching new logs-with-workaround.zip. > Server in cluster hangs during start after previous kill > -------------------------------------------------------- > > Key: WFLY-10736 > URL: https://issues.jboss.org/browse/WFLY-10736 > Project: WildFly > Issue Type: Bug > Components: Clustering > Reporter: Miroslav Novak > Assignee: Paul Ferraro > Priority: Blocker > Labels: blocker-WF14 > Attachments: clusterKilTest.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, standalone-full-ha-1.xml, standalone-full-ha-2.xml > > > There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run. > Test Scenario: > * There are two servers. InQueue is deployed on Node 1, > * OutQueue is deployed on Node 2. Both servers are started. > * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2. > * Node 1 is killed and started again. > * All messages are received from OutQueue deployed on Node 2. > Result: > Node 1 does not start after kill and hangs. There is following exception logged in node 2: > {code} > :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster > 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1] > 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster > 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]} > 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17 > 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17 > 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17 > 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131] > Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131] > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131] > at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) > at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71) > at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540) > at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523) > at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334) > at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85) > at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745) > at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) > at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) > at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) > ... 1 more > Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > ... 1 more > [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1] > 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131] > Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131] > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131] > at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) > at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71) > at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540) > at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523) > at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334) > at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85) > at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745) > at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) > at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) > at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) > ... 1 more > Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1 > at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) > at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131] > ... 1 more > [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1] > {code} > There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000. > Attaching thread dump from node 1 when it hangs during start. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (DROOLS-2512) Add support for multi-file validation on the DMN Validator

by Tibor Zimányi (JIRA)

[ https://issues.jboss.org/browse/DROOLS-2512?page=com.atlassian.jira.plugi... ] Tibor Zimányi updated DROOLS-2512: ---------------------------------- Tester: Tibor Zimányi > Add support for multi-file validation on the DMN Validator > ---------------------------------------------------------- > > Key: DROOLS-2512 > URL: https://issues.jboss.org/browse/DROOLS-2512 > Project: Drools > Issue Type: Enhancement > Components: dmn engine > Affects Versions: 7.7.0.Final > Reporter: Edson Tirelli > Assignee: Matteo Mortari > Fix For: 7.10.0.Final > > > Currently the DMN validator supports only single file validation, but with the implementation of support for "import" we now have to add support for multi-file validation as well. -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

[JBoss JIRA] (WFLY-10759) UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException

by tommaso borgato (JIRA)

tommaso borgato created WFLY-10759: -------------------------------------- Summary: UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException Key: WFLY-10759 URL: https://issues.jboss.org/browse/WFLY-10759 Project: WildFly Issue Type: Bug Components: Clustering, Web (Undertow) Affects Versions: 14.0.0.CR1 Reporter: tommaso borgato Assignee: Paul Ferraro The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. The error was observed on node [dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...] : {noformat} [JBossINF] [0m[31m04:10:53,236 ERROR [io.undertow.servlet.request] (default task-203) UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException [JBossINF] at javax.faces.context.FacesContext.getExceptionHandler(FacesContext.java:304) [JBossINF] at javax.faces.event.ExceptionQueuedEventContext.getListenersForEventClass(ExceptionQueuedEventContext.java:272) [JBossINF] at com.sun.faces.application.ApplicationImpl.invokeComponentListenersFor(ApplicationImpl.java:2230) [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:337) [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:295) [JBossINF] at javax.faces.application.ApplicationWrapper.publishEvent(ApplicationWrapper.java:748) [JBossINF] at com.sun.faces.application.WebappLifecycleListener.requestDestroyed(WebappLifecycleListener.java:175) [JBossINF] at com.sun.faces.config.ConfigureListener.requestDestroyed(ConfigureListener.java:408) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestDestroyed(ApplicationListeners.java:260) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:335) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] [0m[33m04:10:53,236 WARN [org.jboss.weld.Servlet] (default task-203) WELD-000717: Unable to deactivate context org.jboss.weld.module.web.context.http.HttpRequestContextImpl@4a4b8c92 when destroying request HttpServletRequestImpl [ GET /clusterbench/ejbservlet ] [JBossINF] [0m[33m04:10:53,239 WARN [org.jboss.weld.Servlet] (default task-203) WELD-000717: Unable to deactivate context org.jboss.weld.module.web.context.http.HttpSessionContextImpl@1135e4b6 when destroying request HttpServletRequestImpl [ GET /clusterbench/ejbservlet ] [JBossINF] [0m[31m04:10:53,239 ERROR [io.undertow.request] (default task-203) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/ejbservlet request {Cookie=[$Version=0; JSESSIONID=3Nq88HEwcXMUhKudBoFAxcbXXnO6MZB7LCvXTBYZ.dev213; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[dev220:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Mon, 30 Jul 2018 08:10:32 GMT]}}: org.infinispan.commons.CacheException: javax.transaction.SystemException: Unable to rollback transaction [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:113) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:202) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: javax.transaction.SystemException: Unable to rollback transaction [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:135) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:108) [JBossINF] ... 27 more [JBossINF] Caused by: javax.transaction.HeuristicMixedException [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:457) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [JBossINF] ... 28 more [JBossINF] Caused by: javax.transaction.xa.XAException [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [JBossINF] ... 31 more [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:229) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:168) [JBossINF] ... 34 more [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336) [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] ... 1 more [JBossINF] Suppressed: org.infinispan.util.logging.TraceException [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:229) [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:168) [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [JBossINF] at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:108) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:202) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] ... 1 more {noformat} {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} {quote} Please also note that this errors was observed after other errors like the one described [here|https://issues.jboss.org/browse/WFLY-10758] so may be related. {quote} -- This message was sent by Atlassian JIRA (v7.5.0#75005)

7 years, 9 months

1
0
0 / 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

jboss-jira July 2018