[jbossts-issues] [JBoss JIRA] (JBTM-3255) Intermittent failure of CI AS_TESTS in multinode module with error `The port 10090 is already in use`
Ondrej Chaloupka (Jira)
issues at jboss.org
Tue Feb 18 10:52:06 EST 2020
[ https://issues.redhat.com/browse/JBTM-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13975303#comment-13975303 ]
Ondrej Chaloupka commented on JBTM-3255:
----------------------------------------
[~tomjenkinson] agree. I investigated more deeper and what I can see it's really the trouble with the {{RemoteEJBClientStatefulBeanFailoverTestCase}} which does not finish the server properly. As there is long time when the test is inactive then the maven surefire (probably) comes and kills the forked JVM process.
The dump file says
{code}
# Created at 2019-12-04T12:07:17.504
Killing self fork JVM. Received SHUTDOWN command from Maven shutdown hook.
{code}
>From what I investigated the trouble is that server stopping for more than hour and it's not stopped. The test waits for the server to be stopped which never happens. The test is stuck and could not clean resources - aka. it's not able to finish with the second server which is still running and waiting for the command to be stopped.
I uploaded the wildfly {{server.log}}s for the both wildfly servers which are running for this test (https://issues.redhat.com/secure/attachment/12465753/JBTM-3255-occupied-port-10990.zip).
The process seems to be that the test {{RemoteEJBClientStatefulBeanFailoverTestCase}} did some testing, in the middle of the testcase one of the servers should be stopped. In this case it tried to stop the server {{wildfly-1}}. But it failed to be finished. In the {{wildfly-2}} log we can see there is long time pause
{code}
2019-12-04 10:46:20,840 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t1) [Context=RemoteEJBClientStatefulFailoverTestBase.jar/infinispan] ISPN100010: Finished rebalance with members [node-1, node-2], topology id 5
2019-12-04 10:46:21,093 INFO [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0010: Deployed "RemoteEJBClientStatefulFailoverTestBase.jar" (runtime-name : "RemoteEJBClientStatefulFailoverTestBase.jar")
2019-12-04 10:46:21,455 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t1) [Context=RemoteEJBClientStatefulFailoverTestBase.jar/infinispan] ISPN100008: Updating cache members list [node-2], topology id 6
2019-12-04 10:46:21,498 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t1) [Context=client-mappings] ISPN100008: Updating cache members list [node-2], topology id 29
2019-12-04 12:07:36,409 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
2019-12-04 12:07:36,412 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2019-12-04 12:07:36,412 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
2019-12-04 12:07:36,414 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel ejb
2019-12-04 12:07:36,415 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0019: Host default-host stopping
{code}
The server log of {{wildfly-1}} says that it's stopping but it never says {{stopped}} (like is normal that WildFly report line {{WFLYSRV0050: WildFly Full 20.0.0.Beta1-SNAPSHOT (WildFly Core 11.0.0.Beta8) stopped in 230ms}}). The {{wildfly-1}} says only:
{code}
2019-12-04 10:46:21,509 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 35) AMQ151003: resource adaptor stopped
2019-12-04 10:46:22,601 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 35) AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.10.1 [0fd79ee7-1682-11ea-8e07-0242ac11000a] stopped, uptime 19.343 seconds
2019-12-04 10:46:22,602 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0008: Undertow HTTP listener default suspending
2019-12-04 10:46:22,605 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 127.0.0.1:8080
2019-12-04 10:46:22,605 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0004: Undertow 2.0.28.Final stopping
{code}
> Intermittent failure of CI AS_TESTS in multinode module with error `The port 10090 is already in use`
> -----------------------------------------------------------------------------------------------------
>
> Key: JBTM-3255
> URL: https://issues.redhat.com/browse/JBTM-3255
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Testing
> Reporter: Ondrej Chaloupka
> Assignee: Ondrej Chaloupka
> Priority: Minor
> Attachments: JBTM-3255-occupied-port-10990.zip
>
>
> There could be observed intermittent failures of AS_TESTS in Narayana CI where the job fails with error {{The port 10090 is already in use. It means that either the server might be already running or there is another process using port 10090.}}[1]
> The reason seems to be there is probably unfinished application server running from the prior testsuite module.
> Here we talk about the second server which starts the port `10090` as the management port.
> The arquillian starts the `multinode` module and the first test fails with error that the WildFly can't be started as port can't be bound.
> It could be that there is either some process occupying the port `10090` on the Linux machine.
> But the more probable is that the prior test which is in `clustering` as `org.jboss.as.test.clustering.cluster.ejb2.stateful.failover.RemoteEJBClientStatefulBeanFailoverTestCase` was not finished properly.
> The log files does not show any particular trouble though.
> [1]
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.818 s <<< FAILURE! - in org.jboss.as.test.multinode.clientinterceptor.multiple.MultipleClientInterceptorTestCase
> [ERROR] org.jboss.as.test.multinode.clientinterceptor.multiple.MultipleClientInterceptorTestCase Time elapsed: 3.818 s <<< ERROR!
> org.jboss.arquillian.container.spi.client.container.LifecycleException:
> The port 10090 is already in use. It means that either the server might be already running or there is another process using port 10090.
> Managed containers do not support connecting to running server instances due to the possible harmful effect of connecting to the wrong server.
> Please stop server (or another process) before running, change to another type of container (e.g. remote) or use jboss.socket.binding.port-offset variable to change the default port.
> To disable this check and allow Arquillian to connect to a running server, set allowConnectingToRunningServer to true in the container configuration
> at org.jboss.as.arquillian.container.managed.ManagedDeployableContainer.failDueToRunning(ManagedDeployableContainer.java:323)
> at org.jboss.as.arquillian.container.managed.ManagedDeployableContainer.startInternal(ManagedDeployableContainer.java:81)
> at org.jboss.as.arquillian.container.CommonDeployableContainer.start(CommonDeployableContainer.java:123)
> at org.jboss.arquillian.container.impl.ContainerImpl.start(ContainerImpl.java:179)
> at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:137)
> at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:133)
> at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forContainer(ContainerLifecycleController.java:208)
> at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.startContainer(ContainerLifecycleController.java:133)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
> at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
> at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
> at org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createContainerContext(ContainerDeploymentContextHandler.java:54)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
> at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
> at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
> at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
> at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
> at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$2.perform(ContainerLifecycleController.java:70)
> at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$2.perform(ContainerLifecycleController.java:64)
> at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forEachSuiteContainer(ContainerLifecycleController.java:181)
> at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.startSuiteContainers(ContainerLifecycleController.java:64)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
> at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
> at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
> at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
> at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
> at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
> at org.jboss.arquillian.container.test.impl.client.ContainerEventController.execute(ContainerEventController.java:83)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
> at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
> at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
> at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:69)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
> at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
> at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
> at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
> at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.beforeSuite(EventTestRunnerAdaptor.java:71)
> at org.jboss.arquillian.junit.AdaptorManager.initializeAdaptor(AdaptorManager.java:23)
> at org.jboss.arquillian.junit.AdaptorManagerWithNotifier.initializeAdaptor(AdaptorManagerWithNotifier.java:19)
> at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:109)
> at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
> at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
> at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
More information about the jbossts-issues
mailing list