[jboss-jira] [JBoss JIRA] (WFLY-10917) Server fails to shutdown

[ https://issues.jboss.org/browse/WFLY-10917?page=com.atlassian.jira.plugin... ] tommaso borgato updated WFLY-10917: ----------------------------------- Description: Affected scenario is {{[perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt/]}}: it is a 4 nodes fail-over scenario where JGroups communication is encrypted asymmetrically: {noformat} <subsystem xmlns="urn:jboss:domain:jgroups:6.0" default-stack="udp"> <channels default="ee"> <channel name="ee" stack="udp" cluster="ejb"/> </channels> <stacks> <stack name="udp"> <transport type="UDP" socket-binding="jgroups-udp"/> <protocol type="PING"/> <protocol type="MERGE3"/> <protocol type="FD_SOCK"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="ASYM_ENCRYPT" xmlns=""> <property name="encrypt_entire_message">true</property> <property name="sym_keylength">128</property> <property name="sym_algorithm">AES/ECB/PKCS5Padding</property> <property name="asym_keylength">512</property> <property name="asym_algorithm">RSA</property> </protocol> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="AUTH" xmlns=""> <property name="auth_class">org.jgroups.auth.MD5Token</property> <property name="auth_value">MyPassword</property> <property name="token_hash">MD5</property> </protocol> <protocol type="pbcast.GMS"/> <protocol type="UFC"/> <protocol type="MFC"/> <protocol type="FRAG3"/> </stack> <stack name="tcp"> <transport type="TCP" socket-binding="jgroups-tcp"/> <socket-protocol type="MPING" socket-binding="jgroups-mping"/> <protocol type="MERGE3"/> <protocol type="FD_SOCK"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="MFC"/> <protocol type="FRAG3"/> </stack> </stacks> </subsystem> {noformat} The problem is that, after one node of the cluster is shut down and restarted, the remaining nodes fail to shut-down gracefully and their JVM has to be killed. The default shut-down waiting time is 1 minute: it has been extended to 5 but the nodes still fail to shut-down even in 5 minutes. Find attached: - the complete log of one node failing to shut-down (17293.log) - the thread dump of this node just before being killed (after a 5 minutes time-out) (perf20-consoleText.txt) The logs show many messages like the following (as in JGRP-2282): {noformat} [JBossINF] 05:36:35,687 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf21; dropping it [JBossINF] 05:36:36,668 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf21; dropping it [JBossINF] 05:36:36,879 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-14,ejb,perf20) perf20: received message without encrypt header from perf18; dropping it {noformat} was: Affected scenario is {{[perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt/]}}: it is a 4 nodes fail-over scenario where JGroups communication is encrypted asymmetrically: {noformat} <subsystem xmlns="urn:jboss:domain:jgroups:6.0" default-stack="udp"> <channels default="ee"> <channel name="ee" stack="udp" cluster="ejb"/> </channels> <stacks> <stack name="udp"> <transport type="UDP" socket-binding="jgroups-udp"/> <protocol type="PING"/> <protocol type="MERGE3"/> <protocol type="FD_SOCK"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="ASYM_ENCRYPT" xmlns=""> <property name="encrypt_entire_message">true</property> <property name="sym_keylength">128</property> <property name="sym_algorithm">AES/ECB/PKCS5Padding</property> <property name="asym_keylength">512</property> <property name="asym_algorithm">RSA</property> </protocol> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="AUTH" xmlns=""> <property name="auth_class">org.jgroups.auth.MD5Token</property> <property name="auth_value">MyPassword</property> <property name="token_hash">MD5</property> </protocol> <protocol type="pbcast.GMS"/> <protocol type="UFC"/> <protocol type="MFC"/> <protocol type="FRAG3"/> </stack> <stack name="tcp"> <transport type="TCP" socket-binding="jgroups-tcp"/> <socket-protocol type="MPING" socket-binding="jgroups-mping"/> <protocol type="MERGE3"/> <protocol type="FD_SOCK"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="MFC"/> <protocol type="FRAG3"/> </stack> </stacks> </subsystem> {noformat} The problem is that, after one node of the cluster is shut down and restarted, the remaining nodes fail to shut-down gracefully and their JVM has to be killed. The default shut-down waiting time is 1 minute: it has been extended to 5 but the nodes still fail to shut-down even in 5 minutes. Find attached: - the complete log of one node failing to shut-down (17293.log) - the thread dump of this node just before being killed (after a 5 minutes time-out) (perf20-consoleText.txt) -- This message was sent by Atlassian JIRA (v7.5.0#75005)