[jboss-jira] [JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError

[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10758: ----------------------------------- Description: {quote} Please NOTE this error was found when running the job run on a machine segment with UDP problems: I would suggest to use it just to possibly verify the java.lang.StackOverflowError; otherwise I'll close as "Explained"; {quote} The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] 04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] 04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] 04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] 04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] 04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] 04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] 04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] 04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] 04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] 04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] 04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] 04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] 04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] 04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] 04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] 04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] 04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * node dev212 left the cluster: {noformat} [JBossINF] 04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] 04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] 04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] 04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] 04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] 04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] 04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] 04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] 04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed: {noformat} [JBossINF] 04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] 04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] 04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] 04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] 04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] 04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] 04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] 04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] 04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] 04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} {quote} Please also note that when node dev213 left the cluster, node dev212 had not re-joined the cluster yet, or so it looks in the logs. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] 07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] 07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] 07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] 07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] 07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] 07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] 07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] 07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] 07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} was: The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm. h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events: * the server started and the 4 nodes formed the cluster: {noformat} [JBossINF] 04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand) [JBossINF] 04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] 04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] 04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] 04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] 04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] 04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] 04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213] [JBossINF] 04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster [JBossINF] 04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] 04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] 04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] 04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] 04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] 04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster [JBossINF] 04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214] [JBossINF] 04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster {noformat} * node dev212 left the cluster: {noformat} [JBossINF] 04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] 04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] 04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] 04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] 04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] 04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster [JBossINF] 04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214] [JBossINF] 04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster {noformat} * rebalance with remaining nodes was concluded: {noformat} [JBossINF] 04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19 {noformat} And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed: {noformat} [JBossINF] 04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message [JBossINF] 04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] 04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] 04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] 04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] 04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] 04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] 04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214] [JBossINF] 04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster [JBossINF] 04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} the block that is repeated over and over in the stack is the following: {noformat} [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) {noformat} 3396 sampling errors occurred in this run. {quote} Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error. {quote} {quote} Please also note that when node dev213 left the cluster, node dev212 had not re-joined the cluster yet, or so it looks in the logs. {quote} h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...] This run was performed with the following jgroups parameters: {noformat} <protocol type="FD_ALL"> <property name="timeout">5000</property> <property name="interval">1000</property> <property name="timeout_check_interval">2000</property> </protocol> <protocol type="VERIFY_SUSPECT"> <property name="timeout">5000</property> </protocol> {noformat} but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]: {noformat} [JBossINF] 07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] 07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] 07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] 07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] 07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] 07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] 07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214] [JBossINF] 07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster [JBossINF] 07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136) [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113) [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151) [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117) [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853) [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166) [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) ... {noformat} unlike in the previous run, the stack block that is repeated over and over is: {noformat} [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) {noformat} -- This message was sent by Atlassian JIRA (v7.5.0#75005)