[JBoss JIRA] (WFLY-10773) JGRP000029: failed sending message: java.io.IOException: Socket Closed
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10773?page=com.atlassian.jira.plugin... ]
tommaso borgato updated WFLY-10773:
-----------------------------------
Description:
The error was observed in scenario {{*[eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-db-session-tests/job/eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7_JJB/22/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by server shut-down and re-start;
The cluster nodes were configured to use TCP stack for communication:
{code:xml}
<subsystem xmlns="urn:jboss:domain:jgroups:6.0" default-stack="tcp">
<channels default="ee">
<channel name="ee" stack="tcp" cluster="ejb"/>
</channels>
<stacks>
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<socket-protocol type="MPING" socket-binding="jgroups-mping"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
{code}
The 4 cluster nodes store session data into an ivalidation cache backed by a MYSQL Database:
{code:xml}
<invalidation-cache name="offload">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<jdbc-store data-source="testDS" fetch-state="false" passivation="false" purge="false" shared="true" dialect="MYSQL">
<table prefix="s">
<id-column name="id" type="VARCHAR(255)"/>
<data-column name="datum" type="VARBINARY(10000)"/>
<timestamp-column name="version" type="BIGINT"/>
</table>
</jdbc-store>
</invalidation-cache>
{code}
The error was observed on node {{*[dev214|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-db-session-tests/job/eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7_JJB/22/console-dev214/]*}}; here and attempt to isolate the events that may be relevant:
* node dev213 was shut-down and re-started but had not yet re-joined the cluster:
{noformat}
[JBossINF] [0m[0m02:19:07,082 INFO [org.infinispan.CLUSTER] (thread-21,ejb,dev214) ISPN100001: Node dev213 left the cluster
{noformat}
* current node dev214 is initating shut-down:
{noformat}
2018/07/31 02:21:43:593 EDT [INFO ][Thread-88] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:test - [SHUTDOWN] JBossShutdown server host: dev214:9990
{noformat}
* then we observe the error:
{noformat}
[JBossINF] [0m[31m02:21:44,588 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-30,ejb,dev214) JGRP000029: dev214: failed sending message to dev215 (59 bytes): java.io.IOException: Socket Closed, headers: UNICAST3: ACK, seqno=137, conn_id=1, ts=131, TP: [cluster_name=ejb]
{noformat}
* current node dev214 completes shut-down:
{noformat}
2018/07/31 02:21:45:459 EDT [DEBUG][RMI TCP Connection(27)-10.16.91.122] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:test - Server is down.
{noformat}
was:
The error was observed in scenario {{*[eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-db-session-tests/job/eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7_JJB/22/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by server shut-down and re-start;
The cluster nodes were configured to use the TCP stack for communication:
{code:xml}
<subsystem xmlns="urn:jboss:domain:jgroups:6.0" default-stack="tcp">
<channels default="ee">
<channel name="ee" stack="tcp" cluster="ejb"/>
</channels>
<stacks>
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<socket-protocol type="MPING" socket-binding="jgroups-mping"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
{code}
The 4 cluster nodes store session data into an ivalidation cache backed by a MYSQL Database:
{code:xml}
<invalidation-cache name="offload">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<jdbc-store data-source="testDS" fetch-state="false" passivation="false" purge="false" shared="true" dialect="MYSQL">
<table prefix="s">
<id-column name="id" type="VARCHAR(255)"/>
<data-column name="datum" type="VARBINARY(10000)"/>
<timestamp-column name="version" type="BIGINT"/>
</table>
</jdbc-store>
</invalidation-cache>
{code}
The error was observed on node {{*[dev214|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-db-session-tests/job/eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7_JJB/22/console-dev214/]*}}; here and attempt to isolate the events that may be relevant:
* node dev213 was shut-down and re-started but had not yet re-joined the cluster:
{noformat}
[JBossINF] [0m[0m02:19:07,082 INFO [org.infinispan.CLUSTER] (thread-21,ejb,dev214) ISPN100001: Node dev213 left the cluster
{noformat}
* current node dev214 is initating shut-down:
{noformat}
2018/07/31 02:21:43:593 EDT [INFO ][Thread-88] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:test - [SHUTDOWN] JBossShutdown server host: dev214:9990
{noformat}
* then we observe the error:
{noformat}
[JBossINF] [0m[31m02:21:44,588 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-30,ejb,dev214) JGRP000029: dev214: failed sending message to dev215 (59 bytes): java.io.IOException: Socket Closed, headers: UNICAST3: ACK, seqno=137, conn_id=1, ts=131, TP: [cluster_name=ejb]
{noformat}
* current node dev214 completes shut-down:
{noformat}
2018/07/31 02:21:45:459 EDT [DEBUG][RMI TCP Connection(27)-10.16.91.122] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:test - Server is down.
{noformat}
> JGRP000029: failed sending message: java.io.IOException: Socket Closed
> ----------------------------------------------------------------------
>
> Key: WFLY-10773
> URL: https://issues.jboss.org/browse/WFLY-10773
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 14.0.0.CR1
> Reporter: tommaso borgato
> Assignee: Paul Ferraro
>
> The error was observed in scenario {{*[eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-db-session-tests/job/eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7_JJB/22/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by server shut-down and re-start;
> The cluster nodes were configured to use TCP stack for communication:
> {code:xml}
> <subsystem xmlns="urn:jboss:domain:jgroups:6.0" default-stack="tcp">
> <channels default="ee">
> <channel name="ee" stack="tcp" cluster="ejb"/>
> </channels>
> <stacks>
> <stack name="udp">
> <transport type="UDP" socket-binding="jgroups-udp"/>
> <protocol type="PING"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="UFC"/>
> <protocol type="MFC"/>
> <protocol type="FRAG3"/>
> </stack>
> <stack name="tcp">
> <transport type="TCP" socket-binding="jgroups-tcp"/>
> <socket-protocol type="MPING" socket-binding="jgroups-mping"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="MFC"/>
> <protocol type="FRAG3"/>
> </stack>
> </stacks>
> </subsystem>
> {code}
> The 4 cluster nodes store session data into an ivalidation cache backed by a MYSQL Database:
> {code:xml}
> <invalidation-cache name="offload">
> <locking isolation="REPEATABLE_READ"/>
> <transaction mode="BATCH"/>
> <jdbc-store data-source="testDS" fetch-state="false" passivation="false" purge="false" shared="true" dialect="MYSQL">
> <table prefix="s">
> <id-column name="id" type="VARCHAR(255)"/>
> <data-column name="datum" type="VARBINARY(10000)"/>
> <timestamp-column name="version" type="BIGINT"/>
> </table>
> </jdbc-store>
> </invalidation-cache>
> {code}
> The error was observed on node {{*[dev214|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-db-session-tests/job/eap-7x-db-failover-db-session-shutdown-repl-sync-mysql-5-7_JJB/22/console-dev214/]*}}; here and attempt to isolate the events that may be relevant:
> * node dev213 was shut-down and re-started but had not yet re-joined the cluster:
> {noformat}
> [JBossINF] [0m[0m02:19:07,082 INFO [org.infinispan.CLUSTER] (thread-21,ejb,dev214) ISPN100001: Node dev213 left the cluster
> {noformat}
> * current node dev214 is initating shut-down:
> {noformat}
> 2018/07/31 02:21:43:593 EDT [INFO ][Thread-88] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:test - [SHUTDOWN] JBossShutdown server host: dev214:9990
> {noformat}
> * then we observe the error:
> {noformat}
> [JBossINF] [0m[31m02:21:44,588 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-30,ejb,dev214) JGRP000029: dev214: failed sending message to dev215 (59 bytes): java.io.IOException: Socket Closed, headers: UNICAST3: ACK, seqno=137, conn_id=1, ts=131, TP: [cluster_name=ejb]
> {noformat}
> * current node dev214 completes shut-down:
> {noformat}
> 2018/07/31 02:21:45:459 EDT [DEBUG][RMI TCP Connection(27)-10.16.91.122] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:test - Server is down.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (DROOLS-2810) Kie-dmn OSGI feature can not be installed
by Jiri Petrlik (JIRA)
Jiri Petrlik created DROOLS-2810:
------------------------------------
Summary: Kie-dmn OSGI feature can not be installed
Key: DROOLS-2810
URL: https://issues.jboss.org/browse/DROOLS-2810
Project: Drools
Issue Type: Bug
Components: integration
Reporter: Jiri Petrlik
Assignee: Jiri Petrlik
Fix For: 7.10.0.Final
It is not possible to install kie-dmn OSGI feature. Installation fails with following exception:
Unable to resolve org.kie.dmn.core/7.10.0.201807310901: missing requirement [org.kie.dmn.core/7.10.0.201807310901] osgi.wiring.package; filter:="(osgi.wiring.package=org.drools.model)"
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (WFLY-10772) WildFly TS failures on Java 11 (--add-modules java.se)
by Rostislav Svoboda (JIRA)
[ https://issues.jboss.org/browse/WFLY-10772?page=com.atlassian.jira.plugin... ]
Rostislav Svoboda reassigned WFLY-10772:
----------------------------------------
Assignee: Rostislav Svoboda
> WildFly TS failures on Java 11 (--add-modules java.se)
> -------------------------------------------------------
>
> Key: WFLY-10772
> URL: https://issues.jboss.org/browse/WFLY-10772
> Project: WildFly
> Issue Type: Bug
> Components: Test Suite
> Reporter: Rostislav Svoboda
> Assignee: Rostislav Svoboda
>
> WildFly TS failures on Java 11
> Majority of fails coul go away by adding {{--add-modules java.se}} into {{modular.jdk.args}} in jdk9 module
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (WFLY-10758) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10758?page=com.atlassian.jira.plugin... ]
tommaso borgato updated WFLY-10758:
-----------------------------------
Description:
{quote}
Please NOTE this error was found when running the job run on a machine segment with UDP problems: I would suggest to use it just to possibly verify the java.lang.StackOverflowError; otherwise I'll close as "Explained";
{quote}
The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events:
* the server started and the 4 nodes formed the cluster:
{noformat}
[JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand)
[JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
[JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
[JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
[JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
[JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
[JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
[JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
[JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
[JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
[JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
[JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
[JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
[JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
[JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
[JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
[JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
{noformat}
* node dev212 left the cluster:
{noformat}
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
{noformat}
* rebalance with remaining nodes was concluded:
{noformat}
[JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
{noformat}
And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed:
{noformat}
[JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
the block that is repeated over and over in the stack is the following:
{noformat}
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
{noformat}
3396 sampling errors occurred in this run.
{quote}
Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
{quote}
{quote}
Please also note that when node dev213 left the cluster, node dev212 had not re-joined the cluster yet, or so it looks in the logs.
{quote}
h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
This run was performed with the following jgroups parameters:
{noformat}
<protocol type="FD_ALL">
<property name="timeout">5000</property>
<property name="interval">1000</property>
<property name="timeout_check_interval">2000</property>
</protocol>
<protocol type="VERIFY_SUSPECT">
<property name="timeout">5000</property>
</protocol>
{noformat}
but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]:
{noformat}
[JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151)
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
unlike in the previous run, the stack block that is repeated over and over is:
{noformat}
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
{noformat}
was:
The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events:
* the server started and the 4 nodes formed the cluster:
{noformat}
[JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand)
[JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
[JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
[JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
[JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
[JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
[JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
[JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
[JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
[JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
[JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
[JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
[JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
[JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
[JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
[JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
[JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
{noformat}
* node dev212 left the cluster:
{noformat}
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
[JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
{noformat}
* rebalance with remaining nodes was concluded:
{noformat}
[JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
{noformat}
And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed:
{noformat}
[JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
[JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
[JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
the block that is repeated over and over in the stack is the following:
{noformat}
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
{noformat}
3396 sampling errors occurred in this run.
{quote}
Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
{quote}
{quote}
Please also note that when node dev213 left the cluster, node dev212 had not re-joined the cluster yet, or so it looks in the logs.
{quote}
h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
This run was performed with the following jgroups parameters:
{noformat}
<protocol type="FD_ALL">
<property name="timeout">5000</property>
<property name="interval">1000</property>
<property name="timeout_check_interval">2000</property>
</protocol>
<protocol type="VERIFY_SUSPECT">
<property name="timeout">5000</property>
</protocol>
{noformat}
but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]:
{noformat}
[JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
[JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
[JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136)
[JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
[JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151)
[JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117)
[JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
[JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
[JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
[JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
...
{noformat}
unlike in the previous run, the stack block that is repeated over and over is:
{noformat}
[JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
[JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
{noformat}
> ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> -----------------------------------------------------------------------------------------------------
>
> Key: WFLY-10758
> URL: https://issues.jboss.org/browse/WFLY-10758
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 14.0.0.CR1
> Reporter: tommaso borgato
> Assignee: Paul Ferraro
>
> {quote}
> Please NOTE this error was found when running the job run on a machine segment with UDP problems: I would suggest to use it just to possibly verify the java.lang.StackOverflowError; otherwise I'll close as "Explained";
> {quote}
> The error was observed in scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/]*}}: a 4 nodes cluster with a mod_jk load balancer where fail-over is introduced by killing the server jvm.
> h2. [First Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
> The error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the following events:
> * the server started and the 4 nodes formed the cluster:
> {noformat}
> [JBossINF] [0m[0m04:04:32,829 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Beta2-SNAPSHOT (WildFly Core 6.0.0.Alpha4) started in 6051ms - Started 973 of 1160 services (477 services are lazy, passive or on-demand)
> [JBossINF] [0m[0m04:04:38,873 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
> [JBossINF] [0m[0m04:04:38,876 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
> [JBossINF] [0m[0m04:04:38,877 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
> [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
> [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
> [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
> [JBossINF] [0m[0m04:04:38,878 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|2] (3) [dev212, dev215, dev213]
> [JBossINF] [0m[0m04:04:38,879 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev213 joined the cluster
> [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
> [JBossINF] [0m[0m04:04:45,267 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
> [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
> [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
> [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
> [JBossINF] [0m[0m04:04:45,268 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
> [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev215, dev213, dev214]
> [JBossINF] [0m[0m04:04:45,269 INFO [org.infinispan.CLUSTER] (thread-9,ejb,dev215) ISPN100000: Node dev214 joined the cluster
> {noformat}
> * node dev212 left the cluster:
> {noformat}
> [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,724 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,725 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|4] (3) [dev215, dev213, dev214]
> [JBossINF] [0m[0m04:08:13,726 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-29,ejb,dev215) ISPN100001: Node dev212 left the cluster
> {noformat}
> * rebalance with remaining nodes was concluded:
> {noformat}
> [JBossINF] [0m[0m04:08:14,478 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t18) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [dev215, dev213, dev214], topology id 19
> {noformat}
> And then, the error was observed on node {{*[dev215|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_perf27_JJB/1/console-dev215/]*}} after the node ({{*dev213*}}) was failed:
> {noformat}
> [JBossINF] [0m[33m04:10:03,055 WARN [org.jgroups.protocols.UDP] (TQ-Bundler-4,ejb,dev215) JGRP000032: dev215: no physical address for f9eaadfd-8df2-bbee-54e5-7b83f3095679, dropping message
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,213 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN000094: Received new cluster view for channel ejb: [dev215|5] (2) [dev215, dev214]
> [JBossINF] [0m[0m04:10:22,214 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-90,ejb,dev215) ISPN100001: Node dev213 left the cluster
> [JBossINF] [0m[31m04:10:22,404 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t7) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandHeader(ReplicableCommandExternalizer.java:87)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:110)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
> [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> ...
> {noformat}
> the block that is repeated over and over in the stack is the following:
> {noformat}
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> {noformat}
> 3396 sampling errors occurred in this run.
> {quote}
> Please note that this scenario is the counterpart of scenario {{*[eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Clustering_JJB/view/clustering-ejb-ejbservlet-tests/job/eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync_JJB/]*}} with the only difference of being executed on a different segment (perf27 instead of perf17). This other scenario has zero sampling errors and doesn't exhibit this error.
> {quote}
> {quote}
> Please also note that when node dev213 left the cluster, node dev212 had not re-joined the cluster yet, or so it looks in the logs.
> {quote}
> h2. [Second Run|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/E...]
> This run was performed with the following jgroups parameters:
> {noformat}
> <protocol type="FD_ALL">
> <property name="timeout">5000</property>
> <property name="interval">1000</property>
> <property name="timeout_check_interval">2000</property>
> </protocol>
> <protocol type="VERIFY_SUSPECT">
> <property name="timeout">5000</property>
> </protocol>
> {noformat}
> but unluckily the error was observed again on node [dev213|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/vie...]:
> {noformat}
> [JBossINF] [0m[0m07:16:53,977 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,978 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,979 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN000094: Received new cluster view for channel ejb: [dev213|4] (3) [dev213, dev215, dev214]
> [JBossINF] [0m[0m07:16:53,980 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-21,ejb,dev213) ISPN100001: Node dev212 left the cluster
> [JBossINF] [0m[31m07:16:54,160 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.StackOverflowError
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:118)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$Externalizer.writeObject(GlobalTransaction.java:136)
> [JBossINF] at org.infinispan.transaction.xa.GlobalTransaction$AbstractGlobalTxExternalizer.writeObject(GlobalTransaction.java:114)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> [JBossINF] at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.writeTo(AbstractTransactionBoundaryCommand.java:113)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.writeTo(LockControlCommand.java:151)
> [JBossINF] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:121)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:117)
> [JBossINF] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:67)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
> [JBossINF] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1007)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:853)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommand(RpcManagerImpl.java:166)
> [JBossINF] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:192)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:74)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:78)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> ...
> {noformat}
> unlike in the previous run, the stack block that is repeated over and over is:
> {noformat}
> [JBossINF] at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleExceptionOnCommandReturn(BaseStateTransferInterceptor.java:248)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommandReturn(BaseStateTransferInterceptor.java:261)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (WFLY-9804) AuthenticationForwardingSFSFTestCase fails on JDK9
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-9804?page=com.atlassian.jira.plugin.... ]
Darran Lofthouse reassigned WFLY-9804:
--------------------------------------
Assignee: (was: Darran Lofthouse)
> AuthenticationForwardingSFSFTestCase fails on JDK9
> ---------------------------------------------------
>
> Key: WFLY-9804
> URL: https://issues.jboss.org/browse/WFLY-9804
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Affects Versions: 12.0.0.Beta1
> Reporter: Tomaz Cerar
> Labels: jdk9
>
> AuthenticationForwardingSFSFTestCase test fails on JDK9 with
> {noformat}
> javax.naming.CommunicationException: EJBCLIENT000062: Failed to look up "/entry-ejb/EntryBeanSFSB!org.wildfly.test.manual.elytron.seccontext.Entry?stateful" [Root exception is org.jboss.ejb.client.RequestSendFailedException: EJBCLIENT000409: No more destinations are available]
> at org.jboss.ejb.client.EJBRootContext.lookupNative(EJBRootContext.java:160)
> at org.wildfly.naming.client.AbstractContext.lookup(AbstractContext.java:84)
> at org.wildfly.naming.client.WildFlyRootContext.lookup(WildFlyRootContext.java:144)
> at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
> at org.wildfly.test.manual.elytron.seccontext.SeccontextUtil.lookup(SeccontextUtil.java:177)
> at org.wildfly.test.manual.elytron.seccontext.AbstractSecurityContextPropagationTestBase.lambda$getDoubleWhoAmICallable$0(AbstractSecurityContextPropagationTestBase.java:435)
> at org.wildfly.common.context.Contextual.runCallable(Contextual.java:127)
> at org.wildfly.test.manual.elytron.seccontext.AbstractAuthenticationForwardingTestCase.testClientOauthbearerInsufficientRolesFails(AbstractAuthenticationForwardingTestCase.java:154)
> .....
> Caused by: org.jboss.ejb.client.RequestSendFailedException: EJBCLIENT000409: No more destinations are available
> at org.jboss.ejb.client.NamingEJBClientInterceptor.handleSessionCreation(NamingEJBClientInterceptor.java:100)
> at org.jboss.ejb.client.EJBSessionCreationInvocationContext.proceed(EJBSessionCreationInvocationContext.java:70)
> at org.jboss.ejb.client.TransactionInterceptor.handleSessionCreation(TransactionInterceptor.java:100)
> at org.jboss.ejb.client.EJBSessionCreationInvocationContext.proceed(EJBSessionCreationInvocationContext.java:70)
> at org.jboss.ejb.client.EJBClientContext.createSession(EJBClientContext.java:835)
> at org.jboss.ejb.client.EJBClient.createSessionProxy(EJBClient.java:198)
> at org.jboss.ejb.client.EJBRootContext.lookupNative(EJBRootContext.java:158)
> ... 137 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (WFLY-8399) Upgrade jboss-negotiation to 3.0.5
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-8399?page=com.atlassian.jira.plugin.... ]
Darran Lofthouse reassigned WFLY-8399:
--------------------------------------
Assignee: (was: Darran Lofthouse)
> Upgrade jboss-negotiation to 3.0.5
> ----------------------------------
>
> Key: WFLY-8399
> URL: https://issues.jboss.org/browse/WFLY-8399
> Project: WildFly
> Issue Type: Component Upgrade
> Components: Security
> Reporter: Tomas Hofman
> Labels: legacy
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (WFLY-9856) [JDK9+] org.jboss.security.negotiation package is exported by two jars
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-9856?page=com.atlassian.jira.plugin.... ]
Darran Lofthouse reassigned WFLY-9856:
--------------------------------------
Assignee: (was: Darran Lofthouse)
> [JDK9+] org.jboss.security.negotiation package is exported by two jars
> ----------------------------------------------------------------------
>
> Key: WFLY-9856
> URL: https://issues.jboss.org/browse/WFLY-9856
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Reporter: Tomaz Cerar
>
> Currently if you have
> jboss-negotiation-common-3.0.4.Final.jar and jboss-negotiation-extras-3.0.4.Final.jar
> on your module path, jvm complains as both jars export package org.jboss.security.negotiation package
> which violates the modules contract where only one module (jar) can provide single package.
> example error that jvm prints
> {noformat}
> Error: Modules jboss.negotiation.common and jboss.negotiation.extras export package org.jboss.security.negotiation to module wildfly.clustering.common
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (WFCORE-3836) Add operation to read the identities of a realm
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFCORE-3836?page=com.atlassian.jira.plugi... ]
Darran Lofthouse reassigned WFCORE-3836:
----------------------------------------
Assignee: (was: Darran Lofthouse)
> Add operation to read the identities of a realm
> -----------------------------------------------
>
> Key: WFCORE-3836
> URL: https://issues.jboss.org/browse/WFCORE-3836
> Project: WildFly Core
> Issue Type: Feature Request
> Components: Security
> Reporter: Claudio Miranda
>
> The following realm resources contains operations to add, read, remove identity, but there is no operation to read all or a partial list of identities of a realm. It would improve usability on HAL to list all or a partial list of identities.
> The realm resources are: custom-modifiable-realm, filesystem-realm, properties-realm
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (WFLY-9914) Server reload breaks security context
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-9914?page=com.atlassian.jira.plugin.... ]
Darran Lofthouse reassigned WFLY-9914:
--------------------------------------
Assignee: (was: Darran Lofthouse)
> Server reload breaks security context
> -------------------------------------
>
> Key: WFLY-9914
> URL: https://issues.jboss.org/browse/WFLY-9914
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Affects Versions: 11.0.0.Final
> Environment: Ubuntu 16.04 LTS, Oracle JDK 1.8.0_161
> Reporter: Harald Wellmann
>
> h3. Summary
> A minimal example webapp using Soteria and DeltaSpike Security works as expected when first deployed to WildFly.
> After issuing a {{reload}} command via {{jboss-cli.sh}}, the application no longer works, since no groups are associated to the caller principal.
> The problem no longer occurs after a server shutdown and restart.
> h3. Details
> {noformat}
> # Start server
> $ ${JBOSS_HOME}/bin/standalone.sh
> # Build and deploy demo
> $ git clone https://github.com/hwellmann/security-demo.git
> $ cd security-demo
> $ mvn deploy
> # Request protected resource
> $ curl -u operator:secret http://localhost:8080/api/version
> {"version":1}
> # Reload server
> $ ${JBOSS_HOME}/bin/jboss-cli.sh -c --command=:reload
> # Issue same request, access denied
> $ curl -u operator:secret http://localhost:8080/api/version
> {"message":"requested access to the resource is denied"}
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0
[JBoss JIRA] (WFLY-10308) PolicyContext.getContext("javax.security.auth.Subject.container")
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-10308?page=com.atlassian.jira.plugin... ]
Darran Lofthouse reassigned WFLY-10308:
---------------------------------------
Assignee: (was: Darran Lofthouse)
> PolicyContext.getContext("javax.security.auth.Subject.container")
> -----------------------------------------------------------------
>
> Key: WFLY-10308
> URL: https://issues.jboss.org/browse/WFLY-10308
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Reporter: Lin Gao
>
> PolicyContext.getContext("javax.security.auth.Subject.container") returns null when using elytron to protect a web application.
> This should return the authenticated subject.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 4 months
- 1
- 0