[JBoss JIRA] (WFCORE-3943) Embedded server operations executed using different classloaders
by Yeray Borges (JIRA)
Yeray Borges created WFCORE-3943:
------------------------------------
Summary: Embedded server operations executed using different classloaders
Key: WFCORE-3943
URL: https://issues.jboss.org/browse/WFCORE-3943
Project: WildFly Core
Issue Type: Bug
Components: Embedded
Reporter: Yeray Borges
Assignee: Yeray Borges
The embedded server, when it is used by Galleon, launched via client/jboss-cli-client.jar or in the test suite, is using the application classloader. The embedded server when is launched via CLI uses the {{org.jboss.as.cli}} module classloader.
Running the embedded server using inconsistent classloaders could lead to different bugs and strange behaviors, for example, we are using different implementations of XMLStreamWriter when the embedded server is executed via CLI and via Galleon.
Specifically, this issue cannot be reproduced with the test suite, because the woodstock XMLStreamWriter is already in the classpath.
As suggested by [~brian.stansberry], we should use {{org.jboss.as.controller}} classloader to run the embedded server operations.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (JGRP-2253) FD_SOCK is not working in AWS environment
by Sibin Karnavar (JIRA)
[ https://issues.jboss.org/browse/JGRP-2253?page=com.atlassian.jira.plugin.... ]
Sibin Karnavar commented on JGRP-2253:
--------------------------------------
Detecting Via FD was slow because it takes more time, around 13 seconds
<FD timeout="3000" max_tries="3" />
<VERIFY_SUSPECT timeout="3000" />
You will see in my logs, it took 13 seconds for printing VERIFY_SUSPECT.
To reduce the time, FD_SOCK was efficient but did not work well while terminating instances. Works really good when kill or stop e.t.c
> FD_SOCK is not working in AWS environment
> -----------------------------------------
>
> Key: JGRP-2253
> URL: https://issues.jboss.org/browse/JGRP-2253
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.10
> Environment: AWS - EC2
> Reporter: Sibin Karnavar
> Assignee: Bela Ban
> Fix For: 4.0.13
>
>
> We have our failure detection defined like below.
> <FD_SOCK external_port="7804" />
> <FD timeout="3000" max_tries="3" />
> <VERIFY_SUSPECT timeout="3000" />
> Please note that we have used FD instead of FD_ALL in AWS. We will be changing it to FD_ALL later after detailed testing.
> In my local, this is working perfect. As soon as I kill my node, I was able to see that view change was happening immediately with FD_SOCK.
> We were not mentioning the external_port in the FD_SOCK but later I thought it may be an issue with the port and defined it as 7804 and added the same port to the security group that allows to access this port among all the nodes. So no issue with the port.
> Can you please let us know if we need any additional configurations to make FD_SOCK works well in AWS.
> Thanks,
> Sibin
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (JGRP-2253) FD_SOCK is not working in AWS environment
by Sibin Karnavar (JIRA)
[ https://issues.jboss.org/browse/JGRP-2253?page=com.atlassian.jira.plugin.... ]
Sibin Karnavar commented on JGRP-2253:
--------------------------------------
And I thought the suspect messages in my logs are due to FD fails 3 times and I have defined VERIFY_SUSPECT. So What I was mentioning was some how FD_SOCK is not detecting TCP connection breakage while terminating an EC2 instance but FD is catching it before FD_SOCK. But FD_SOCK is able to detect it when I do a kill -9.
> FD_SOCK is not working in AWS environment
> -----------------------------------------
>
> Key: JGRP-2253
> URL: https://issues.jboss.org/browse/JGRP-2253
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.10
> Environment: AWS - EC2
> Reporter: Sibin Karnavar
> Assignee: Bela Ban
> Fix For: 4.0.13
>
>
> We have our failure detection defined like below.
> <FD_SOCK external_port="7804" />
> <FD timeout="3000" max_tries="3" />
> <VERIFY_SUSPECT timeout="3000" />
> Please note that we have used FD instead of FD_ALL in AWS. We will be changing it to FD_ALL later after detailed testing.
> In my local, this is working perfect. As soon as I kill my node, I was able to see that view change was happening immediately with FD_SOCK.
> We were not mentioning the external_port in the FD_SOCK but later I thought it may be an issue with the port and defined it as 7804 and added the same port to the security group that allows to access this port among all the nodes. So no issue with the port.
> Can you please let us know if we need any additional configurations to make FD_SOCK works well in AWS.
> Thanks,
> Sibin
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (JGRP-2253) FD_SOCK is not working in AWS environment
by Sibin Karnavar (JIRA)
[ https://issues.jboss.org/browse/JGRP-2253?page=com.atlassian.jira.plugin.... ]
Sibin Karnavar commented on JGRP-2253:
--------------------------------------
<<One thing you should do is to separate TCP.bind_port (7803), TCP.diagnostics_port (7806) and FD_SOCK.start_port (7804)>>
Currently we have defined my bind port as 7803 and port_range as 0. So There is no chance that a member can use 7804 as my TCP bind port. We did this because we want to make sure only one port is dedicated for TCP bind port. In addition to that, we have only one member running in one EC2 instance. Do you think it is still an issue? if So What is the other best way to separate it out.
<<Note that I've run JGroups on AWS before, and I didn't define any start_port or client_bind_port in FD_SOCK, so these ports are picked by the OS.>>
We have defined a security group inbound ports in AWS so only certain port is allows to access an EC2 instance. If JGroup uses a random client_bind_port, then it won't work. That's why we have mentioned a specific port. I was able to make FD_SOCK work after defining that port. Thats why kill -9 always work. But for some reason terminating EC2 instance from console doesn't work well.
> FD_SOCK is not working in AWS environment
> -----------------------------------------
>
> Key: JGRP-2253
> URL: https://issues.jboss.org/browse/JGRP-2253
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.10
> Environment: AWS - EC2
> Reporter: Sibin Karnavar
> Assignee: Bela Ban
> Fix For: 4.0.13
>
>
> We have our failure detection defined like below.
> <FD_SOCK external_port="7804" />
> <FD timeout="3000" max_tries="3" />
> <VERIFY_SUSPECT timeout="3000" />
> Please note that we have used FD instead of FD_ALL in AWS. We will be changing it to FD_ALL later after detailed testing.
> In my local, this is working perfect. As soon as I kill my node, I was able to see that view change was happening immediately with FD_SOCK.
> We were not mentioning the external_port in the FD_SOCK but later I thought it may be an issue with the port and defined it as 7804 and added the same port to the security group that allows to access this port among all the nodes. So no issue with the port.
> Can you please let us know if we need any additional configurations to make FD_SOCK works well in AWS.
> Thanks,
> Sibin
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (WFLY-10621) Server stop indefinitely blocked by DistributableSessionManager in lifecycleLock.writeLockInterruptibly()
by Petr H (JIRA)
[ https://issues.jboss.org/browse/WFLY-10621?page=com.atlassian.jira.plugin... ]
Petr H updated WFLY-10621:
--------------------------
Summary: Server stop indefinitely blocked by DistributableSessionManager in lifecycleLock.writeLockInterruptibly() (was: Server stop indefinitely blocked on by DistributableSessionManager in lifecycleLock.writeLockInterruptibly())
> Server stop indefinitely blocked by DistributableSessionManager in lifecycleLock.writeLockInterruptibly()
> ---------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10621
> URL: https://issues.jboss.org/browse/WFLY-10621
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 11.0.0.Final
> Environment: CentOS 7.4.1708
> Java 1.8.0_172-b11
> WildFly 11.0.0.Final
> Reporter: Petr H
> Assignee: Paul Ferraro
> Labels: clustering, sessions
> Attachments: threaddump-wfly11-stop-blocked.txt
>
>
> Note: This is report for the issue reported here first https://developer.jboss.org/thread/277883
> It can be seen at WildFly 11 and EAP 7.1 at least, however as the particular code in the DistributableSessionManager class seems to be unchanged even in WildFly 13 it's likely to occur there as well. (We test WildFly 11 because it's closest to the EAP 7.1 which is being considered for the production use later in this project)
> Stop server operation may hang indefinitely due to the thread such as the following one being blocked:
> {noformat}
> "ServerService Thread Pool -- 24" #49 prio=5 os_prio=0 tid=0x00000000035d1000 nid=0x4de1 waiting on condition [0x00007f4bddc46000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c4e61998> (a java.util.concurrent.locks.StampedLock)
> at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
> at java.util.concurrent.locks.StampedLock.writeLockInterruptibly(StampedLock.java:413)
> at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.stop(DistributableSessionManager.java:90)
> - locked <0x00000000c4e60090> (a org.wildfly.clustering.web.undertow.session.DistributableSessionManager)
> at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:599)
> at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:589)
> at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
> at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$1005/749195997.call(Unknown Source)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
> at io.undertow.servlet.core.DeploymentManagerImpl.stop(DeploymentManagerImpl.java:603)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.stopContext(UndertowDeploymentService.java:139)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$2.run(UndertowDeploymentService.java:120)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {noformat}
> The related code in the DistributableSessionManager makes use of the [java.util.concurrent.locks.StampedLock.writeLockInterruptibly()|https://d...] method which can theoretically block indefinitely and thus prevent the server stop from ever completing.
> The change was introduced in the following commit [WFLY-7695 Block session manager stop until all sessions are closed|https://github.com/wildfly/wildfly/commit/94863deeb96c4536ec5e8266...]
> This can occur even in non-clustered environments (non-HA profile) with web applications just being marked as distributable (web.xml).
> We don't know what prevents the lock from being obtained (unable to determine from the thread dump) so we can't unfortunately do much about it at the moment from the application side.
> We only tried to change the infinispan web local-cache configuration due to _ISPN000136_ errors intermittedly occurring, but that seems to have no effect. We tried various combinations of *locking isolation* (REPEATABLE_READ, READ_COMMITED), *transaction locking* (OPTIMISTIC, PESSIMISTIC) and *transaction mode* (NONE, BATCH) settings and it seems to occur with all of them.
> Note: The issue doesn't occur always, only in some cases.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (WFLY-10621) Server stop indefinitely blocked on by DistributableSessionManager in lifecycleLock.writeLockInterruptibly()
by Petr H (JIRA)
[ https://issues.jboss.org/browse/WFLY-10621?page=com.atlassian.jira.plugin... ]
Petr H updated WFLY-10621:
--------------------------
Description:
Note: This is report for the issue reported here first https://developer.jboss.org/thread/277883
It can be seen at WildFly 11 and EAP 7.1 at least, however as the particular code in the DistributableSessionManager class seems to be unchanged even in WildFly 13 it's likely to occur there as well. (We test WildFly 11 because it's closest to the EAP 7.1 which is being considered for the production use later in this project)
Stop server operation may hang indefinitely due to the thread such as the following one being blocked:
{noformat}
"ServerService Thread Pool -- 24" #49 prio=5 os_prio=0 tid=0x00000000035d1000 nid=0x4de1 waiting on condition [0x00007f4bddc46000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c4e61998> (a java.util.concurrent.locks.StampedLock)
at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
at java.util.concurrent.locks.StampedLock.writeLockInterruptibly(StampedLock.java:413)
at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.stop(DistributableSessionManager.java:90)
- locked <0x00000000c4e60090> (a org.wildfly.clustering.web.undertow.session.DistributableSessionManager)
at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:599)
at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:589)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$1005/749195997.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at io.undertow.servlet.core.DeploymentManagerImpl.stop(DeploymentManagerImpl.java:603)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.stopContext(UndertowDeploymentService.java:139)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$2.run(UndertowDeploymentService.java:120)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{noformat}
The related code in the DistributableSessionManager makes use of the [java.util.concurrent.locks.StampedLock.writeLockInterruptibly()|https://d...] method which can theoretically block indefinitely and thus prevent the server stop from ever completing.
The change was introduced in the following commit [WFLY-7695 Block session manager stop until all sessions are closed|https://github.com/wildfly/wildfly/commit/94863deeb96c4536ec5e8266...]
This can occur even in non-clustered environments (non-HA profile) with web applications just being marked as distributable (web.xml).
We don't know what prevents the lock from being obtained (unable to determine from the thread dump) so we can't unfortunately do much about it at the moment from the application side.
We only tried to change the infinispan web local-cache configuration due to _ISPN000136_ errors intermittedly occurring, but that seems to have no effect. We tried various combinations of *locking isolation* (REPEATABLE_READ, READ_COMMITED), *transaction locking* (OPTIMISTIC, PESSIMISTIC) and *transaction mode* (NONE, BATCH) settings and it seems to occur with all of them.
Note: The issue doesn't occur always, only in some cases.
was:
Note: This is report for the issue reported here first https://developer.jboss.org/thread/277883
It can be seen at WildFly 11 and EAP 7.1 at least, however as the particular code in the DistributableSessionManager class seems to be unchanged even in WildFly 13 it's likely to occur there as well. (We test WildFly 11 because it's closest to the EAP 7.1 which is being considered for the production use later in this project)
Stop server operation may hang indefinitely due to the thread such as the following one being blocked:
{noformat}
"ServerService Thread Pool -- 24" #49 prio=5 os_prio=0 tid=0x00000000035d1000 nid=0x4de1 waiting on condition [0x00007f4bddc46000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c4e61998> (a java.util.concurrent.locks.StampedLock)
at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
at java.util.concurrent.locks.StampedLock.writeLockInterruptibly(StampedLock.java:413)
at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.stop(DistributableSessionManager.java:90)
- locked <0x00000000c4e60090> (a org.wildfly.clustering.web.undertow.session.DistributableSessionManager)
at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:599)
at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:589)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$1005/749195997.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at io.undertow.servlet.core.DeploymentManagerImpl.stop(DeploymentManagerImpl.java:603)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.stopContext(UndertowDeploymentService.java:139)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$2.run(UndertowDeploymentService.java:120)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{noformat}
The related code in the DistributableSessionManager makes use of the *java.util.concurrent.locks.StampedLock.writeLockInterruptibly* method which can theoretically block indefinitely and thus prevent the server stop from ever completing.
The change was introduced in the following commit [WFLY-7695 Block session manager stop until all sessions are closed|https://github.com/wildfly/wildfly/commit/94863deeb96c4536ec5e8266...]
This can occur even in non-clustered environments (non-HA profile) with web applications just being marked as distributable (web.xml).
We don't know what prevents the lock from being obtained (unable to determine from the thread dump) so we can't unfortunately do much about it at the moment from the application side.
We only tried to change the infinispan web local-cache configuration due to _ISPN000136_ errors intermittedly occurring, but that seems to have no effect. We tried various combinations of *locking isolation* (REPEATABLE_READ, READ_COMMITED), *transaction locking* (OPTIMISTIC, PESSIMISTIC) and *transaction mode* (NONE, BATCH) settings and it seems to occur with all of them.
Note: The issue doesn't occur always, only in some cases.
> Server stop indefinitely blocked on by DistributableSessionManager in lifecycleLock.writeLockInterruptibly()
> ------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10621
> URL: https://issues.jboss.org/browse/WFLY-10621
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 11.0.0.Final
> Environment: CentOS 7.4.1708
> Java 1.8.0_172-b11
> WildFly 11.0.0.Final
> Reporter: Petr H
> Assignee: Paul Ferraro
> Labels: clustering, sessions
> Attachments: threaddump-wfly11-stop-blocked.txt
>
>
> Note: This is report for the issue reported here first https://developer.jboss.org/thread/277883
> It can be seen at WildFly 11 and EAP 7.1 at least, however as the particular code in the DistributableSessionManager class seems to be unchanged even in WildFly 13 it's likely to occur there as well. (We test WildFly 11 because it's closest to the EAP 7.1 which is being considered for the production use later in this project)
> Stop server operation may hang indefinitely due to the thread such as the following one being blocked:
> {noformat}
> "ServerService Thread Pool -- 24" #49 prio=5 os_prio=0 tid=0x00000000035d1000 nid=0x4de1 waiting on condition [0x00007f4bddc46000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c4e61998> (a java.util.concurrent.locks.StampedLock)
> at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
> at java.util.concurrent.locks.StampedLock.writeLockInterruptibly(StampedLock.java:413)
> at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.stop(DistributableSessionManager.java:90)
> - locked <0x00000000c4e60090> (a org.wildfly.clustering.web.undertow.session.DistributableSessionManager)
> at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:599)
> at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:589)
> at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
> at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$1005/749195997.call(Unknown Source)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
> at io.undertow.servlet.core.DeploymentManagerImpl.stop(DeploymentManagerImpl.java:603)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.stopContext(UndertowDeploymentService.java:139)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$2.run(UndertowDeploymentService.java:120)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {noformat}
> The related code in the DistributableSessionManager makes use of the [java.util.concurrent.locks.StampedLock.writeLockInterruptibly()|https://d...] method which can theoretically block indefinitely and thus prevent the server stop from ever completing.
> The change was introduced in the following commit [WFLY-7695 Block session manager stop until all sessions are closed|https://github.com/wildfly/wildfly/commit/94863deeb96c4536ec5e8266...]
> This can occur even in non-clustered environments (non-HA profile) with web applications just being marked as distributable (web.xml).
> We don't know what prevents the lock from being obtained (unable to determine from the thread dump) so we can't unfortunately do much about it at the moment from the application side.
> We only tried to change the infinispan web local-cache configuration due to _ISPN000136_ errors intermittedly occurring, but that seems to have no effect. We tried various combinations of *locking isolation* (REPEATABLE_READ, READ_COMMITED), *transaction locking* (OPTIMISTIC, PESSIMISTIC) and *transaction mode* (NONE, BATCH) settings and it seems to occur with all of them.
> Note: The issue doesn't occur always, only in some cases.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (WFLY-10621) Server stop indefinitely blocked on by DistributableSessionManager in lifecycleLock.writeLockInterruptibly()
by Petr H (JIRA)
Petr H created WFLY-10621:
-----------------------------
Summary: Server stop indefinitely blocked on by DistributableSessionManager in lifecycleLock.writeLockInterruptibly()
Key: WFLY-10621
URL: https://issues.jboss.org/browse/WFLY-10621
Project: WildFly
Issue Type: Bug
Components: Clustering
Affects Versions: 11.0.0.Final
Environment: CentOS 7.4.1708
Java 1.8.0_172-b11
WildFly 11.0.0.Final
Reporter: Petr H
Assignee: Paul Ferraro
Attachments: threaddump-wfly11-stop-blocked.txt
Note: This is report for the issue reported here first https://developer.jboss.org/thread/277883
It can be seen at WildFly 11 and EAP 7.1 at least, however as the particular code in the DistributableSessionManager class seems to be unchanged even in WildFly 13 it's likely to occur there as well. (We test WildFly 11 because it's closest to the EAP 7.1 which is being considered for the production use later in this project)
Stop server operation may hang indefinitely due to the thread such as the following one being blocked:
{noformat}
"ServerService Thread Pool -- 24" #49 prio=5 os_prio=0 tid=0x00000000035d1000 nid=0x4de1 waiting on condition [0x00007f4bddc46000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c4e61998> (a java.util.concurrent.locks.StampedLock)
at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
at java.util.concurrent.locks.StampedLock.writeLockInterruptibly(StampedLock.java:413)
at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.stop(DistributableSessionManager.java:90)
- locked <0x00000000c4e60090> (a org.wildfly.clustering.web.undertow.session.DistributableSessionManager)
at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:599)
at io.undertow.servlet.core.DeploymentManagerImpl$3.call(DeploymentManagerImpl.java:589)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$1005/749195997.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1006/1165195653.call(Unknown Source)
at io.undertow.servlet.core.DeploymentManagerImpl.stop(DeploymentManagerImpl.java:603)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.stopContext(UndertowDeploymentService.java:139)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$2.run(UndertowDeploymentService.java:120)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{noformat}
The related code in the DistributableSessionManager makes use of the *java.util.concurrent.locks.StampedLock.writeLockInterruptibly* method which can theoretically block indefinitely and thus prevent the server stop from ever completing.
The change was introduced in the following commit [WFLY-7695 Block session manager stop until all sessions are closed|https://github.com/wildfly/wildfly/commit/94863deeb96c4536ec5e8266...]
This can occur even in non-clustered environments (non-HA profile) with web applications just being marked as distributable (web.xml).
We don't know what prevents the lock from being obtained (unable to determine from the thread dump) so we can't unfortunately do much about it at the moment from the application side.
We only tried to change the infinispan web local-cache configuration due to _ISPN000136_ errors intermittedly occurring, but that seems to have no effect. We tried various combinations of *locking isolation* (REPEATABLE_READ, READ_COMMITED), *transaction locking* (OPTIMISTIC, PESSIMISTIC) and *transaction mode* (NONE, BATCH) settings and it seems to occur with all of them.
Note: The issue doesn't occur always, only in some cases.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years