[
https://issues.redhat.com/browse/JBTM-3255?page=com.atlassian.jira.plugin...
]
Ondrej Chaloupka commented on JBTM-3255:
----------------------------------------
[~tomjenkinson] agree. I investigated more deeper and what I can see it's really the
trouble with the {{RemoteEJBClientStatefulBeanFailoverTestCase}} which does not finish the
server properly. As there is long time when the test is inactive then the maven surefire
(probably) comes and kills the forked JVM process.
The dump file says
{code}
# Created at 2019-12-04T12:07:17.504
Killing self fork JVM. Received SHUTDOWN command from Maven shutdown hook.
{code}
From what I investigated the trouble is that server stopping for more
than hour and it's not stopped. The test waits for the server to be stopped which
never happens. The test is stuck and could not clean resources - aka. it's not able to
finish with the second server which is still running and waiting for the command to be
stopped.
I uploaded the wildfly {{server.log}}s for the both wildfly servers which are running for
this test
(
https://issues.redhat.com/secure/attachment/12465753/JBTM-3255-occupied-p...).
The process seems to be that the test {{RemoteEJBClientStatefulBeanFailoverTestCase}} did
some testing, in the middle of the testcase one of the servers should be stopped. In this
case it tried to stop the server {{wildfly-1}}. But it failed to be finished. In the
{{wildfly-2}} log we can see there is long time pause
{code}
2019-12-04 10:46:20,840 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t1)
[Context=RemoteEJBClientStatefulFailoverTestBase.jar/infinispan] ISPN100010: Finished
rebalance with members [node-1, node-2], topology id 5
2019-12-04 10:46:21,093 INFO [org.jboss.as.server] (management-handler-thread - 1)
WFLYSRV0010: Deployed "RemoteEJBClientStatefulFailoverTestBase.jar"
(runtime-name : "RemoteEJBClientStatefulFailoverTestBase.jar")
2019-12-04 10:46:21,455 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t1)
[Context=RemoteEJBClientStatefulFailoverTestBase.jar/infinispan] ISPN100008: Updating
cache members list [node-2], topology id 6
2019-12-04 10:46:21,498 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t1)
[Context=client-mappings] ISPN100008: Updating cache members list [node-2], topology id
29
2019-12-04 12:07:36,409 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4)
WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
2019-12-04 12:07:36,412 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2019-12-04 12:07:36,412 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
2019-12-04 12:07:36,414 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel ejb
2019-12-04 12:07:36,415 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1)
WFLYUT0019: Host default-host stopping
{code}
The server log of {{wildfly-1}} says that it's stopping but it never says {{stopped}}
(like is normal that WildFly report line {{WFLYSRV0050: WildFly Full 20.0.0.Beta1-SNAPSHOT
(WildFly Core 11.0.0.Beta8) stopped in 230ms}}). The {{wildfly-1}} says only:
{code}
2019-12-04 10:46:21,509 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool
-- 35) AMQ151003: resource adaptor stopped
2019-12-04 10:46:22,601 INFO [org.apache.activemq.artemis.core.server] (ServerService
Thread Pool -- 35) AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.10.1
[0fd79ee7-1682-11ea-8e07-0242ac11000a] stopped, uptime 19.343 seconds
2019-12-04 10:46:22,602 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1)
WFLYUT0008: Undertow HTTP listener default suspending
2019-12-04 10:46:22,605 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1)
WFLYUT0007: Undertow HTTP listener default stopped, was bound to 127.0.0.1:8080
2019-12-04 10:46:22,605 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7)
WFLYUT0004: Undertow 2.0.28.Final stopping
{code}
Intermittent failure of CI AS_TESTS in multinode module with error
`The port 10090 is already in use`
-----------------------------------------------------------------------------------------------------
Key: JBTM-3255
URL:
https://issues.redhat.com/browse/JBTM-3255
Project: JBoss Transaction Manager
Issue Type: Bug
Components: Testing
Reporter: Ondrej Chaloupka
Assignee: Ondrej Chaloupka
Priority: Minor
Attachments: JBTM-3255-occupied-port-10990.zip
There could be observed intermittent failures of AS_TESTS in Narayana CI where the job
fails with error {{The port 10090 is already in use. It means that either the server might
be already running or there is another process using port 10090.}}[1]
The reason seems to be there is probably unfinished application server running from the
prior testsuite module.
Here we talk about the second server which starts the port `10090` as the management
port.
The arquillian starts the `multinode` module and the first test fails with error that the
WildFly can't be started as port can't be bound.
It could be that there is either some process occupying the port `10090` on the Linux
machine.
But the more probable is that the prior test which is in `clustering` as
`org.jboss.as.test.clustering.cluster.ejb2.stateful.failover.RemoteEJBClientStatefulBeanFailoverTestCase`
was not finished properly.
The log files does not show any particular trouble though.
[1]
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.818 s
<<< FAILURE! - in
org.jboss.as.test.multinode.clientinterceptor.multiple.MultipleClientInterceptorTestCase
[ERROR]
org.jboss.as.test.multinode.clientinterceptor.multiple.MultipleClientInterceptorTestCase
Time elapsed: 3.818 s <<< ERROR!
org.jboss.arquillian.container.spi.client.container.LifecycleException:
The port 10090 is already in use. It means that either the server might be already
running or there is another process using port 10090.
Managed containers do not support connecting to running server instances due to the
possible harmful effect of connecting to the wrong server.
Please stop server (or another process) before running, change to another type of
container (e.g. remote) or use jboss.socket.binding.port-offset variable to change the
default port.
To disable this check and allow Arquillian to connect to a running server, set
allowConnectingToRunningServer to true in the container configuration
at
org.jboss.as.arquillian.container.managed.ManagedDeployableContainer.failDueToRunning(ManagedDeployableContainer.java:323)
at
org.jboss.as.arquillian.container.managed.ManagedDeployableContainer.startInternal(ManagedDeployableContainer.java:81)
at
org.jboss.as.arquillian.container.CommonDeployableContainer.start(CommonDeployableContainer.java:123)
at org.jboss.arquillian.container.impl.ContainerImpl.start(ContainerImpl.java:179)
at
org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:137)
at
org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:133)
at
org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forContainer(ContainerLifecycleController.java:208)
at
org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.startContainer(ContainerLifecycleController.java:133)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
at
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
at
org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createContainerContext(ContainerDeploymentContextHandler.java:54)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
at
org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$2.perform(ContainerLifecycleController.java:70)
at
org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$2.perform(ContainerLifecycleController.java:64)
at
org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forEachSuiteContainer(ContainerLifecycleController.java:181)
at
org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.startSuiteContainers(ContainerLifecycleController.java:64)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
at
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
at
org.jboss.arquillian.container.test.impl.client.ContainerEventController.execute(ContainerEventController.java:83)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
at
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
at
org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:69)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
at
org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.beforeSuite(EventTestRunnerAdaptor.java:71)
at org.jboss.arquillian.junit.AdaptorManager.initializeAdaptor(AdaptorManager.java:23)
at
org.jboss.arquillian.junit.AdaptorManagerWithNotifier.initializeAdaptor(AdaptorManagerWithNotifier.java:19)
at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:109)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
{code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)