[jboss-jira] [JBoss JIRA] (WFLY-10917) Server fails to shutdown
tommaso borgato (JIRA)
issues at jboss.org
Fri Aug 24 09:09:00 EDT 2018
[ https://issues.jboss.org/browse/WFLY-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tommaso borgato updated WFLY-10917:
-----------------------------------
Description:
Affected scenario is {{[perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt/]}}: it is a 4 nodes fail-over scenario where JGroups communication is encrypted asymmetrically:
{noformat}
<subsystem xmlns="urn:jboss:domain:jgroups:6.0" default-stack="udp">
<channels default="ee">
<channel name="ee" stack="udp" cluster="ejb"/>
</channels>
<stacks>
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="ASYM_ENCRYPT" xmlns="">
<property name="encrypt_entire_message">true</property>
<property name="sym_keylength">128</property>
<property name="sym_algorithm">AES/ECB/PKCS5Padding</property>
<property name="asym_keylength">512</property>
<property name="asym_algorithm">RSA</property>
</protocol>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="AUTH" xmlns="">
<property name="auth_class">org.jgroups.auth.MD5Token</property>
<property name="auth_value">MyPassword</property>
<property name="token_hash">MD5</property>
</protocol>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<socket-protocol type="MPING" socket-binding="jgroups-mping"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
{noformat}
The problem is that, after one node of the cluster is shut down and restarted, the remaining nodes fail to shut-down gracefully and their JVM has to be killed.
The default shut-down waiting time is 1 minute: it has been extended to 5 but the nodes still fail to shut-down even in 5 minutes.
Find attached:
- the complete log of one node failing to shut-down (17293.log)
- the thread dump of this node just before being killed (after a 5 minutes time-out) (perf20-consoleText.txt)
The server logs messages like the following (as in JGRP-2282):
{noformat}
[JBossINF] [0m[31m05:36:35,687 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf21; dropping it
[JBossINF] [0m[31m05:36:36,668 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf21; dropping it
[JBossINF] [0m[31m05:36:36,879 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf18; dropping it
{noformat}
before failing to shut-down.
was:
Affected scenario is {{[perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt/]}}: it is a 4 nodes fail-over scenario where JGroups communication is encrypted asymmetrically:
{noformat}
<subsystem xmlns="urn:jboss:domain:jgroups:6.0" default-stack="udp">
<channels default="ee">
<channel name="ee" stack="udp" cluster="ejb"/>
</channels>
<stacks>
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="ASYM_ENCRYPT" xmlns="">
<property name="encrypt_entire_message">true</property>
<property name="sym_keylength">128</property>
<property name="sym_algorithm">AES/ECB/PKCS5Padding</property>
<property name="asym_keylength">512</property>
<property name="asym_algorithm">RSA</property>
</protocol>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="AUTH" xmlns="">
<property name="auth_class">org.jgroups.auth.MD5Token</property>
<property name="auth_value">MyPassword</property>
<property name="token_hash">MD5</property>
</protocol>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<socket-protocol type="MPING" socket-binding="jgroups-mping"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
{noformat}
The problem is that, after one node of the cluster is shut down and restarted, the remaining nodes fail to shut-down gracefully and their JVM has to be killed.
The default shut-down waiting time is 1 minute: it has been extended to 5 but the nodes still fail to shut-down even in 5 minutes.
Find attached:
- the complete log of one node failing to shut-down (17293.log)
- the thread dump of this node just before being killed (after a 5 minutes time-out) (perf20-consoleText.txt)
The logs show many messages like the following (as in JGRP-2282):
{noformat}
[JBossINF] [0m[31m05:36:35,687 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf21; dropping it
[JBossINF] [0m[31m05:36:36,668 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf21; dropping it
[JBossINF] [0m[31m05:36:36,879 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf18; dropping it
{noformat}
> Server fails to shutdown
> -------------------------
>
> Key: WFLY-10917
> URL: https://issues.jboss.org/browse/WFLY-10917
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 14.0.0.CR1
> Reporter: tommaso borgato
> Assignee: Paul Ferraro
> Priority: Critical
> Attachments: 17293.log, perf20-consoleText.txt
>
>
> Affected scenario is {{[perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt/]}}: it is a 4 nodes fail-over scenario where JGroups communication is encrypted asymmetrically:
> {noformat}
> <subsystem xmlns="urn:jboss:domain:jgroups:6.0" default-stack="udp">
> <channels default="ee">
> <channel name="ee" stack="udp" cluster="ejb"/>
> </channels>
> <stacks>
> <stack name="udp">
> <transport type="UDP" socket-binding="jgroups-udp"/>
> <protocol type="PING"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="ASYM_ENCRYPT" xmlns="">
> <property name="encrypt_entire_message">true</property>
> <property name="sym_keylength">128</property>
> <property name="sym_algorithm">AES/ECB/PKCS5Padding</property>
> <property name="asym_keylength">512</property>
> <property name="asym_algorithm">RSA</property>
> </protocol>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="AUTH" xmlns="">
> <property name="auth_class">org.jgroups.auth.MD5Token</property>
> <property name="auth_value">MyPassword</property>
> <property name="token_hash">MD5</property>
> </protocol>
> <protocol type="pbcast.GMS"/>
> <protocol type="UFC"/>
> <protocol type="MFC"/>
> <protocol type="FRAG3"/>
> </stack>
> <stack name="tcp">
> <transport type="TCP" socket-binding="jgroups-tcp"/>
> <socket-protocol type="MPING" socket-binding="jgroups-mping"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="MFC"/>
> <protocol type="FRAG3"/>
> </stack>
> </stacks>
> </subsystem>
> {noformat}
> The problem is that, after one node of the cluster is shut down and restarted, the remaining nodes fail to shut-down gracefully and their JVM has to be killed.
> The default shut-down waiting time is 1 minute: it has been extended to 5 but the nodes still fail to shut-down even in 5 minutes.
> Find attached:
> - the complete log of one node failing to shut-down (17293.log)
> - the thread dump of this node just before being killed (after a 5 minutes time-out) (perf20-consoleText.txt)
> The server logs messages like the following (as in JGRP-2282):
> {noformat}
> [JBossINF] [0m[31m05:36:35,687 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf21; dropping it
> [JBossINF] [0m[31m05:36:36,668 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf21; dropping it
> [JBossINF] [0m[31m05:36:36,879 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf18; dropping it
> {noformat}
> before failing to shut-down.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
More information about the jboss-jira
mailing list