[jboss-jira] [JBoss JIRA] (WFLY-5806) IllegalStateException during failover of HA Singleton deployment

IllegalStateException during failover of HA Singleton deployment ---------------------------------------------------------------- Key: WFLY-5806 URL: https://issues.jboss.org/browse/WFLY-5806 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 10.0.0.CR4 Reporter: Michal Vinkler Assignee: Paul Ferraro Priority: Minor Fix For: 10.0.0.CR5 Seen in failover tests - HA Singleton deployment scenarios - no matter what failover type was used (graceful shutdown, jvmkill, undeploy). When the node is elected to operate as the singleton provider, sometimes this INFO message is logged: {code} INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-singleton-jbossall.ear, performing full redeploy instead. {code} Then the redeploy will fail: {code} [JBossINF] [0m[33m05:41:07,118 WARN [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-passivating.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException [JBossINF] at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47) [JBossINF] at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) {code} Then these events follows: - This node will no longer operate as the singleton provider - immediately after that the node is re-elected as the singleton provider and the deployment starts successfully See the full stacktrace: {code} [JBossINF] [0m[0m05:41:07,109 INFO [org.wildfly.clustering.server] (OOB-19,ee,perf19) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service [JBossINF] [0m[0m05:41:07,112 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-singleton-jbossall.ear, performing full redeploy instead. [JBossINF] [0m[0m05:41:07,114 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-web-passivating.war, performing full redeploy instead. [JBossINF] [0m[0m05:41:07,114 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-ejb.jar, performing full redeploy instead. [JBossINF] [0m[0m05:41:07,114 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-web-default.war, performing full redeploy instead. [JBossINF] [0m[33m05:41:07,118 WARN [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-passivating.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException [JBossINF] at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47) [JBossINF] at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] [JBossINF] [0m[33m05:41:07,117 WARN [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-default.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException [JBossINF] at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47) [JBossINF] at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] [JBossINF] [0m[33m05:41:07,116 WARN [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-ejb.jar".FIRST_MODULE_USE.service: java.lang.IllegalStateException [JBossINF] at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47) [JBossINF] at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] [JBossINF] [0m[0m05:41:07,150 INFO [org.wildfly.clustering.server] (OOB-19,ee,perf19) WFLYCLSV0002: This node will no longer operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service [JBossINF] [0m[0m05:41:07,170 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel hibernate [JBossINF] [0m[0m05:41:07,171 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000082: Stopping the RpcDispatcher for channel hibernate [JBossINF] [0m[0m05:41:07,171 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting JGroups channel web [JBossINF] [0m[0m05:41:07,172 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher for channel web [JBossINF] [0m[0m05:41:07,172 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-ejb.jar) in 58ms [JBossINF] [0m[0m05:41:07,173 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel ejb [JBossINF] [0m[0m05:41:07,173 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel ejb [JBossINF] [0m[0m05:41:07,176 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-default.war) in 62ms [JBossINF] [0m[0m05:41:07,175 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-passivating.war) in 61ms [JBossINF] [0m[0m05:41:07,184 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment clusterbench-ee7-singleton-jbossall.ear (runtime-name: clusterbench-ee7-singleton-jbossall.ear) in 70ms [JBossINF] [0m[0m05:41:07,186 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "clusterbench-ee7-singleton-jbossall.ear" (runtime-name: "clusterbench-ee7-singleton-jbossall.ear") [JBossINF] [0m[0m05:41:07,187 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 82) WFLYCLINF0003: Stopped default cache from server container [JBossINF] [0m[0m05:41:07,190 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel server [JBossINF] [0m[0m05:41:07,190 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel server [JBossINF] [0m[0m05:41:07,232 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-passivating.war") [JBossINF] [0m[0m05:41:07,232 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-default.war") [JBossINF] [0m[0m05:41:07,232 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-ejb.jar") [JBossINF] [0m[33m05:41:07,294 WARN [org.jboss.as.dependency.private] (MSC service thread 1-8) WFLYSRV0018: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-default.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice. [JBossINF] [0m[33m05:41:07,295 WARN [org.jboss.as.dependency.unsupported] (MSC service thread 1-8) WFLYSRV0019: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-default.war" is using an unsupported module ("org.jgroups:main") which may be changed or removed in future versions without notice. [JBossINF] [0m[33m05:41:07,295 WARN [org.jboss.as.dependency.private] (MSC service thread 1-1) WFLYSRV0018: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-passivating.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice. [JBossINF] [0m[33m05:41:07,295 WARN [org.jboss.as.dependency.unsupported] (MSC service thread 1-1) WFLYSRV0019: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-passivating.war" is using an unsupported module ("org.jgroups:main") which may be changed or removed in future versions without notice. [JBossINF] [0m[0m05:41:07,423 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000078: Starting JGroups channel server [JBossINF] [0m[0m05:41:07,424 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000094: Received new cluster view for channel server: [perf21|13] (4) [perf21, perf18, perf20, perf19] [JBossINF] [0m[0m05:41:07,424 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000079: Channel server local address is perf19, physical addresses are [172.19.1.3:55200] [JBossINF] [0m[0m05:41:07,426 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel web [JBossINF] [0m[0m05:41:07,427 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel web: [perf21|13] (4) [perf21, perf18, perf20, perf19] [JBossINF] [0m[0m05:41:07,427 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel web local address is perf19, physical addresses are [172.19.1.3:55200] [JBossINF] [0m[0m05:41:07,442 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel ejb [JBossINF] [0m[0m05:41:07,444 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel ejb: [perf21|13] (4) [perf21, perf18, perf20, perf19] [JBossINF] [0m[0m05:41:07,444 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel ejb local address is perf19, physical addresses are [172.19.1.3:55200] [JBossINF] [0m[0m05:41:07,444 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel hibernate [JBossINF] [0m[0m05:41:07,445 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel hibernate: [perf21|13] (4) [perf21, perf18, perf20, perf19] [JBossINF] [0m[0m05:41:07,446 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel hibernate local address is perf19, physical addresses are [172.19.1.3:55200] [JBossINF] [0m[0m05:41:07,510 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 82) WFLYCLINF0002: Started default cache from server container [JBossINF] [0m[0m05:41:10,357 INFO [org.wildfly.clustering.server] (OOB-20,ee,perf19) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service [JBossINF] [0m[0m05:41:10,362 INFO [org.jboss.weld.deployer] (MSC service thread 1-6) WFLYWELD0003: Processing weld deployment clusterbench-ee7-singleton-jbossall.ear [JBossINF] [0m[0m05:41:10,384 INFO [org.jboss.weld.deployer] (MSC service thread 1-6) WFLYWELD0003: Processing weld deployment clusterbench-ee7-ejb.jar [JBossINF] [0m[0m05:41:10,387 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'RemoteStatelessSBImpl' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows: [JBossINF] [JBossINF] java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB [JBossINF] java:app/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB [JBossINF] java:module/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB [JBossINF] java:jboss/exported/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB [JBossINF] java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl [JBossINF] java:app/clusterbench-ee7-ejb/RemoteStatelessSBImpl [JBossINF] java:module/RemoteStatelessSBImpl [JBossINF] [JBossINF] [0m[0m05:41:10,387 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'RemoteStatefulSBImpl' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows: [JBossINF] [JBossINF] java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB [JBossINF] java:app/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB [JBossINF] java:module/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB [JBossINF] java:jboss/exported/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB [JBossINF] java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl [JBossINF] java:app/clusterbench-ee7-ejb/RemoteStatefulSBImpl [JBossINF] java:module/RemoteStatefulSBImpl [JBossINF] [JBossINF] [0m[0m05:41:10,387 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'LocalStatefulSB' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows: [JBossINF] [JBossINF] java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB [JBossINF] java:app/clusterbench-ee7-ejb/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB [JBossINF] java:module/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB [JBossINF] java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/LocalStatefulSB [JBossINF] java:app/clusterbench-ee7-ejb/LocalStatefulSB [JBossINF] java:module/LocalStatefulSB [JBossINF] {code} Link: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-singl...