[JBoss JIRA] (WFLY-10531) Wildfly leaks ActiveMQ connections
by Marcel Šebek (JIRA)
[ https://issues.jboss.org/browse/WFLY-10531?page=com.atlassian.jira.plugin... ]
Marcel Šebek commented on WFLY-10531:
-------------------------------------
I've found a possibly related issue https://issues.jboss.org/browse/JBJCA-1358. In wildfly 13, there is an upgrade of ironjacamar from 1.4.7 to 1.4.9, and 1.4.8 contains a fix for the linked issue. We use container-managed JMS context extensively, sometimes with XA transactions. Unfortunately, just replacing ironjacamar module direcotry in wildfly 13 with the one from version 12 doesn't work, so I cannot simply verify whether this fix is the culprit.
I'm ready to try anything you suggest me, or provide any details.
> Wildfly leaks ActiveMQ connections
> ----------------------------------
>
> Key: WFLY-10531
> URL: https://issues.jboss.org/browse/WFLY-10531
> Project: WildFly
> Issue Type: Bug
> Affects Versions: 13.0.0.Final
> Environment: openjdk 8 / openjdk 9, Linux
> Reporter: Marcel Šebek
> Assignee: Jeff Mesnil
>
> After upgrading our application from wildfly 12 to 13, the app started to crash after a while (hours, days, depending on circumstances). It crashes on
> IJ000453: Unable to get managed connection for java:/JmsXA
> and other errors (it simply cannot perform all the jobs it contains). I found that when shutting down the server which has been running for a while, I can see a bunch of these messages in the log:
> WARN [org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri] (ServerService Thread Pool -- 117) [:::] IJ000615: Destroying active connection in pool: ActiveMQConnectionDefinition (org.apache.activemq.artemis.ra.ActiveMQRAManagedConnection@2f37f69)
> Bascially, the longer the server was running, more of these messages are shown. I cannot find a way how to reproduce the issue. When the server runs for short time but with some load, no connection is leaked (or just one, rarely). On the other side, it leaks connections even without any particularly high load (just a few requests and @Schedule jobs) when running for longer time.
> It may also be a bug in our application, which just happen to have more serious impact with the new wildfly version.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10531) Wildfly leaks ActiveMQ connections
by Jason Greene (JIRA)
[ https://issues.jboss.org/browse/WFLY-10531?page=com.atlassian.jira.plugin... ]
Jason Greene reassigned WFLY-10531:
-----------------------------------
Assignee: Jeff Mesnil (was: Jason Greene)
> Wildfly leaks ActiveMQ connections
> ----------------------------------
>
> Key: WFLY-10531
> URL: https://issues.jboss.org/browse/WFLY-10531
> Project: WildFly
> Issue Type: Bug
> Affects Versions: 13.0.0.Final
> Environment: openjdk 8 / openjdk 9, Linux
> Reporter: Marcel Šebek
> Assignee: Jeff Mesnil
>
> After upgrading our application from wildfly 12 to 13, the app started to crash after a while (hours, days, depending on circumstances). It crashes on
> IJ000453: Unable to get managed connection for java:/JmsXA
> and other errors (it simply cannot perform all the jobs it contains). I found that when shutting down the server which has been running for a while, I can see a bunch of these messages in the log:
> WARN [org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri] (ServerService Thread Pool -- 117) [:::] IJ000615: Destroying active connection in pool: ActiveMQConnectionDefinition (org.apache.activemq.artemis.ra.ActiveMQRAManagedConnection@2f37f69)
> Bascially, the longer the server was running, more of these messages are shown. I cannot find a way how to reproduce the issue. When the server runs for short time but with some load, no connection is leaked (or just one, rarely). On the other side, it leaks connections even without any particularly high load (just a few requests and @Schedule jobs) when running for longer time.
> It may also be a bug in our application, which just happen to have more serious impact with the new wildfly version.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10531) Wildfly leaks ActiveMQ connections
by Marcel Šebek (JIRA)
Marcel Šebek created WFLY-10531:
-----------------------------------
Summary: Wildfly leaks ActiveMQ connections
Key: WFLY-10531
URL: https://issues.jboss.org/browse/WFLY-10531
Project: WildFly
Issue Type: Bug
Affects Versions: 13.0.0.Final
Environment: openjdk 8 / openjdk 9, Linux
Reporter: Marcel Šebek
Assignee: Jason Greene
After upgrading our application from wildfly 12 to 13, the app started to crash after a while (hours, days, depending on circumstances). It crashes on
IJ000453: Unable to get managed connection for java:/JmsXA
and other errors (it simply cannot perform all the jobs it contains). I found that when shutting down the server which has been running for a while, I can see a bunch of these messages in the log:
WARN [org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri] (ServerService Thread Pool -- 117) [:::] IJ000615: Destroying active connection in pool: ActiveMQConnectionDefinition (org.apache.activemq.artemis.ra.ActiveMQRAManagedConnection@2f37f69)
Bascially, the longer the server was running, more of these messages are shown. I cannot find a way how to reproduce the issue. When the server runs for short time but with some load, no connection is leaked (or just one, rarely). On the other side, it leaks connections even without any particularly high load (just a few requests and @Schedule jobs) when running for longer time.
It may also be a bug in our application, which just happen to have more serious impact with the new wildfly version.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10530) Domain test failures on IBM8 - IllegalStateException: The LogManager was not properly installed
by Petr Kremensky (JIRA)
Petr Kremensky created WFLY-10530:
-------------------------------------
Summary: Domain test failures on IBM8 - IllegalStateException: The LogManager was not properly installed
Key: WFLY-10530
URL: https://issues.jboss.org/browse/WFLY-10530
Project: WildFly
Issue Type: Bug
Components: Test Suite
Reporter: Petr Kremensky
org.jboss.as.test.integration.domain.suites.DeploymentOverlayTestCase#testDeploymentOverlayInDomainMode and org.jboss.as.test.integration.domain.suites.ReadEnvironmentVariablesTestCase#testReadEnvironmentVariablesForServers from domain module of Wildfly testsuite fails on the latest IBM 8 jdk with java.lang.IllegalStateException: The LogManager was not properly installed (you must set the "java.util.logging.manager" system property to "org.jboss.logmanager.LogManager")
*reproduce*
{noformat}
wildfly/testsuite]$ mvn test -DtestLogToFile=false -pl domain -Dts.domain -Dtest=ReadEnvironmentVariablesTestCase,DeploymentOverlayTestCase
...
[ERROR] Errors:
[ERROR] DeploymentOverlayTestCase.testDeploymentOverlayInDomainMode:300->performHttpCall:340 » NoClassDefFound
[ERROR] ReadEnvironmentVariablesTestCase.testReadEnvironmentVariablesForServers:109->getEnvironmentVariables:148 » ExceptionInInitializer
[INFO]
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0
{noformat}
*Failure message*
{noformat}
java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:316)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:329)
at org.apache.http.conn.ssl.DefaultHostnameVerifier.<init>(DefaultHostnameVerifier.java:72)
at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:955)
at org.apache.http.impl.client.HttpClients.createDefault(HttpClients.java:58)
at org.jboss.as.test.integration.domain.suites.ReadEnvironmentVariablesTestCase.getEnvironmentVariables(ReadEnvironmentVariablesTestCase.java:148)
at org.jboss.as.test.integration.domain.suites.ReadEnvironmentVariablesTestCase.testReadEnvironmentVariablesForServers(ReadEnvironmentVariablesTestCase.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: java.lang.IllegalStateException: The LogManager was not properly installed (you must set the "java.util.logging.manager" system property to "org.jboss.logmanager.LogManager")
at org.jboss.logmanager.Logger.getLogger(Logger.java:57)
at org.apache.commons.logging.JBossLogFactory.<init>(JBossLogFactory.java:48)
at org.apache.commons.logging.LogFactory$Holder.<clinit>(LogFactory.java:42)
... 44 more
{noformat}
*Environment*
{noformat}
$ mvn -version
Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z)
Maven home: /qa/tools/opt/maven3-latest
Java version: 1.8.0_171, vendor: IBM Corporation
Java home: /qa/tools/opt/x86_64/ibm-java-x86_64-sdk-8.0-5.15/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-862.el7.x86_64", arch: "amd64", family: "unix"
---
$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 8.0.5.15 - pxa6480sr5fp15-20180502_01(SR5 FP15))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64 Compressed References 20180425_385365 (JIT enabled, AOT enabled)
OpenJ9 - a7ffbfe
OMR - a531219
IBM - 59ef3dc)
JCL - 20180425_01 based on Oracle jdk8u171-b11
{noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10529) ParseAndMarshalModelsTestCase fails on the latest IBM8 with Failed to register MBean with MBeanServer
by Petr Kremensky (JIRA)
Petr Kremensky created WFLY-10529:
-------------------------------------
Summary: ParseAndMarshalModelsTestCase fails on the latest IBM8 with Failed to register MBean with MBeanServer
Key: WFLY-10529
URL: https://issues.jboss.org/browse/WFLY-10529
Project: WildFly
Issue Type: Bug
Components: Test Suite
Reporter: Petr Kremensky
org.jboss.as.test.manualmode.parse.ParseAndMarshalModelsTestCase from manualmode module of wildlfy testsuite fails with the latest IBM 8 JDK bits.
*reproduce*
{noformat}
wildfly/testsuite]$ mvn test -DtestLogToFile=false -pl integration/manualmode -Dts.manualmode -Dtest=ParseAndMarshalModelsTestCase
...
[ERROR] Tests run: 40, Failures: 40, Errors: 0, Skipped: 0
{noformat}
*Error message appearing in the logs*
{noformat}
Jun 07, 2018 3:57:14 PM org.jboss.msc.service.ServiceContainerImpl <init>
ERROR: MSC000010: Failed to register MBean with MBeanServer
javax.management.InstanceAlreadyExistsException: jboss.msc:type=container,name=host-controller
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:449)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1910)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:978)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:912)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:336)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:534)
at org.jboss.msc.service.ServiceContainerImpl.<init>(ServiceContainerImpl.java:368)
at org.jboss.msc.service.ServiceContainer$Factory.create(ServiceContainer.java:221)
at org.wildfly.core.embedded.EmbeddedHostControllerBootstrap$ShutdownHook.register(EmbeddedHostControllerBootstrap.java:90)
at org.wildfly.core.embedded.EmbeddedHostControllerBootstrap$ShutdownHook.access$100(EmbeddedHostControllerBootstrap.java:80)
at org.wildfly.core.embedded.EmbeddedHostControllerBootstrap.<init>(EmbeddedHostControllerBootstrap.java:54)
at org.wildfly.core.embedded.EmbeddedHostControllerFactory$HostControllerImpl.start(EmbeddedHostControllerFactory.java:270)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.wildfly.core.embedded.EmbeddedManagedProcessImpl.invokeOnServer(EmbeddedManagedProcessImpl.java:88)
at org.wildfly.core.embedded.EmbeddedManagedProcessImpl.start(EmbeddedManagedProcessImpl.java:58)
at org.jboss.as.cli.embedded.EmbedHostControllerHandler.doHandle(EmbedHostControllerHandler.java:268)
at org.jboss.as.cli.handlers.CommandHandlerWithHelp.handle(CommandHandlerWithHelp.java:89)
at org.jboss.as.cli.impl.CommandExecutor$2.lambda$build$0(CommandExecutor.java:685)
at org.jboss.as.cli.impl.CommandExecutor$2$$Lambda$79.00000000200064E0.execute(Unknown Source)
at org.jboss.as.cli.impl.CommandExecutor.lambda$execute$0(CommandExecutor.java:708)
at org.jboss.as.cli.impl.CommandExecutor$$Lambda$78.00000000AD3CF710.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:277)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:811)
{noformat}
*Failure cause*
{noformat}
Caused by: org.jboss.as.cli.CommandLineException: Cannot start embedded server
at org.jboss.as.cli.embedded.EmbedServerHandler.doHandle(EmbedServerHandler.java:300)
at org.jboss.as.cli.handlers.CommandHandlerWithHelp.handle(CommandHandlerWithHelp.java:89)
at org.jboss.as.cli.impl.CommandExecutor$2.lambda$build$0(CommandExecutor.java:685)
at org.jboss.as.cli.impl.CommandExecutor$2$$Lambda$79.00000000200064E0.execute(Unknown Source)
at org.jboss.as.cli.impl.CommandExecutor.lambda$execute$0(CommandExecutor.java:708)
at org.jboss.as.cli.impl.CommandExecutor$$Lambda$78.00000000AD3CF710.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:277)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:811)
Caused by: java.lang.ClassCastException: org.jboss.as.controller.client.helpers.DelegatingModelControllerClient incompatible with org.jboss.as.controller.client.ModelControllerClient
at org.wildfly.core.embedded.EmbeddedManagedProcessImpl.getModelControllerClient(EmbeddedManagedProcessImpl.java:72)
at org.jboss.as.cli.embedded.EmbedServerHandler.doHandle(EmbedServerHandler.java:244)
... 9 more
{noformat}
*Environment*
{noformat}
$ mvn -version
Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z)
Maven home: /qa/tools/opt/maven3-latest
Java version: 1.8.0_171, vendor: IBM Corporation
Java home: /qa/tools/opt/x86_64/ibm-java-x86_64-sdk-8.0-5.15/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-862.el7.x86_64", arch: "amd64", family: "unix"
---
$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 8.0.5.15 - pxa6480sr5fp15-20180502_01(SR5 FP15))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64 Compressed References 20180425_385365 (JIT enabled, AOT enabled)
OpenJ9 - a7ffbfe
OMR - a531219
IBM - 59ef3dc)
JCL - 20180425_01 based on Oracle jdk8u171-b11
{noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month