Re: [jboss-as7-dev] ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

It probably wasn't using the timeout setting added to as7/testsuite/integration/pom.xml. I changed the o.j.a.j.JMXConnectorService default to ten seconds and made it through the integration tests. If we change the default timeout to ten seconds (for a few days), that might cause test failures on busy/slower machines. On 10/20/2011 03:21 PM, Scott Marlow wrote: > No, I still see "java.lang.OutOfMemoryError: unable to create new native > thread". I'll try a shorter timeout, maybe ten seconds instead of thirty. > > Another workaround would be reducing the number of tests in > testsuite/integration (at least until ARQ-632 is fixed). > > On 10/20/2011 02:58 PM, Scott Marlow wrote: >> I'll give that a try. >> >> On 10/20/2011 02:36 PM, Brian Stansberry wrote: >>> I'm not seeing this failure, so it's hard to test a workaround. But if >>> someone who is seeing it can give >>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139... >>> >>> a spin and let me know, I'd appreciate it. >>> >>> Probably easiest is just to check out and build >>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak >>> >>> On 10/20/11 11:49 AM, Brian Stansberry wrote: >>>> See [1] for a good discussion of this; last post points at a possible >>>> workaround. >>>> >>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966 >>>> On 10/19/11 6:26 PM, Stuart Douglas wrote: >>>>> It is probably to do with the number of tests in the test suite, as >>>>> now all the tests that were previously in preview and spec are >>>>> executed in one run. >>>>> >>>>> Stuart >>>>> >>>>> >>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote: >>>>> >>>>>> I agree that we should be closing the connection >>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/j... >>>>>> >>>>>> >>>>>> is an example of where we handled that in AS6). Probably after we >>>>>> are >>>>>> done with the MBeanServerConnection. >>>>>> >>>>>> I created ARQ-632 for this. >>>>>> >>>>>> I was expecting the cause to be a regression but instead perhaps its >>>>>> the performance improvements (we are running unit tests quicker, >>>>>> causing more "jmx server connection timeout" threads to be created). >>>>>> Or, due to the number of tests in the testsuite now. >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Brian Stansberry"<brian.stansberry(a)redhat.com> >>>>>> To: jboss-as7-dev(a)lists.jboss.org >>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM >>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I >>>>>> synced with master yesterday >>>>>> >>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection() >>>>>> >>>>>> >>>>>> >>>>>> is creating a javax.management.remote.JMXConnector but I don't see >>>>>> anything closing it. I'm not sure, but perhaps one of those is >>>>>> created >>>>>> per test method execution? >>>>>> >>>>>> I believe the "JMX server connection timeout 552" threads are >>>>>> created by >>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can >>>>>> see the >>>>>> relevant code in the stack trace. A quick look at that code leads >>>>>> me to >>>>>> believe those threads will by default live for 120 secs after the >>>>>> last >>>>>> request on the connection. >>>>>> >>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote: >>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote: >>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote: >>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun >>>>>>>>> 1.6.0_24-b07. >>>>>>>>> >>>>>>>>> Jesper also got the failures with jdk 6 u26. >>>>>>>>> >>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try >>>>>>>>> that >>>>>>>>> or we could start comparing environment settings. >>>>>>>>> >>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m >>>>>>>>> -XX:MaxPermSize=128m" >>>>>>>>> >>>>>>>>> I don't have that set locally. I'll try that next... >>>>>>>> >>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set. >>>>>>>> >>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that >>>>>>>> makes >>>>>>>> a diff. >>>>>>> >>>>>>> Same failure still (with MAVEN_OPTS set as above and using >>>>>>> jdk1.6.0_24). >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Scott >>>>>>>>> >>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote: >>>>>>>>>> I don't know about the test issues that Ondrej is running into, >>>>>>>>>> but the >>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : >>>>>>>>>> Cannot >>>>>>>>>> create native thread) were encountered once by Carlo (or on our >>>>>>>>>> Hudson >>>>>>>>>> instance, don't remember). But I haven't seen them after >>>>>>>>>> that. It >>>>>>>>>> might >>>>>>>>>> be environment specific. What JDK vendor/version? And are you >>>>>>>>>> running >>>>>>>>>> into this always? >>>>>>>>>> >>>>>>>>>> -Jaikiran >>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: >>>>>>>>>>> How did these testsuite failures slip in? From my own >>>>>>>>>>> experience >>>>>>>>>>> running the updated testsuite, it looks like we now have an >>>>>>>>>>> allTests >>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we >>>>>>>>>>> merged some >>>>>>>>>>> changes and only ran the allTests profile. I think that we >>>>>>>>>>> need to >>>>>>>>>>> continue to use -DallTests instead (that includes the >>>>>>>>>>> integration >>>>>>>>>>> tests). Does anyone know more? >>>>>>>>>>> >>>>>>>>>>> I'm not sure why some people aren't seeing the failures but it >>>>>>>>>>> seems we >>>>>>>>>>> need to track backwards from the symptoms (listed below) or >>>>>>>>>>> rollback >>>>>>>>>>> some changes until it goes away. >>>>>>>>>>> >>>>>>>>>>> If anyone is working on tracking the cause down, please speak >>>>>>>>>>> up. I can >>>>>>>>>>> jump in but don't want to duplicate effort. >>>>>>>>>>> >>>>>>>>>>> Scott >>>>>>>>>>> >>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote: >>>>>>>>>>>> The hudson AS7 test just finished with all integration tests >>>>>>>>>>>> passing. >>>>>>>>>>>> >>>>>>>>>>>> When you say that your using a fresh master, is that a new AS7 >>>>>>>>>>>> source >>>>>>>>>>>> tree or an existing one freshly synced with AS7 master? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>>>>>>>>>>>> I have a fresh master and testsuite runs fine. >>>>>>>>>>>>>> Except 2 failures (for which I also seek comments): >>>>>>>>>>>>>> >>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> server.log: >>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949 >>>>>>>>>>>>> >>>>>>>>>>>>> I didn't see that failure. >>>>>>>>>>>>> >>>>>>>>>>>>> Which folder are you starting the unit test from and what is >>>>>>>>>>>>> your >>>>>>>>>>>>> command line? >>>>>>>>>>>>> >>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the >>>>>>>>>>>>> as7/testsuite/integration folder. >>>>>>>>>>>>> >>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test >>>>>>>>>>>>> (was using >>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed >>>>>>>>>>>>> that but >>>>>>>>>>>>> the test is still in queue. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition >>>>>>>>>>>>>> caused by >>>>>>>>>>>>>> improper use of deployer API, but giving it some time after >>>>>>>>>>>>>> deploy >>>>>>>>>>>>>> didn't help. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ondra >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run >>>>>>>>>>>>>>> fine for a >>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new >>>>>>>>>>>>>>> native thread" >>>>>>>>>>>>>>> errors start showing up in the server.log. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection >>>>>>>>>>>>>>> timeout NNN" >>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a >>>>>>>>>>>>>>> "mvn clean >>>>>>>>>>>>>>> install -DallTests". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Scott >>> >>> >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev