[jboss-as7-dev] ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

Thu Oct 20 17:42:01 EDT 2011

Can you try this:

https://github.com/bstansberry/jboss-as/commit/a3dc2b2730c6c0aedf36257470d0ea8281802088

It's the right thing to do anyway, as it should improve testsuite perf a 
fair bit by not opening connections all the time.

On 10/20/11 3:43 PM, Brian Stansberry wrote:
> I think the AS/ARQ integration (ManagementClient class) can easily
> enough cache and re-use the JMXContext and thus the
> MBeanServerConnection+JMXConnector. I'll send you a patch that does that
> in a bit.
>
> On 10/20/11 3:22 PM, Scott Marlow wrote:
>> It probably wasn't using the timeout setting added to
>> as7/testsuite/integration/pom.xml. I changed the
>> o.j.a.j.JMXConnectorService default to ten seconds and made it through
>> the integration tests.
>>
>> If we change the default timeout to ten seconds (for a few days), that
>> might cause test failures on busy/slower machines.
>>
>> On 10/20/2011 03:21 PM, Scott Marlow wrote:
>>> No, I still see "java.lang.OutOfMemoryError: unable to create new native
>>> thread". I'll try a shorter timeout, maybe ten seconds instead of
>>> thirty.
>>>
>>> Another workaround would be reducing the number of tests in
>>> testsuite/integration (at least until ARQ-632 is fixed).
>>>
>>> On 10/20/2011 02:58 PM, Scott Marlow wrote:
>>>> I'll give that a try.
>>>>
>>>> On 10/20/2011 02:36 PM, Brian Stansberry wrote:
>>>>> I'm not seeing this failure, so it's hard to test a workaround. But if
>>>>> someone who is seeing it can give
>>>>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
>>>>>
>>>>>
>>>>> a spin and let me know, I'd appreciate it.
>>>>>
>>>>> Probably easiest is just to check out and build
>>>>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>>>>>
>>>>> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>>>>>> See [1] for a good discussion of this; last post points at a possible
>>>>>> workaround.
>>>>>>
>>>>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>>>>>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>>>>>> It is probably to do with the number of tests in the test suite, as
>>>>>>> now all the tests that were previously in preview and spec are
>>>>>>> executed in one run.
>>>>>>>
>>>>>>> Stuart
>>>>>>>
>>>>>>>
>>>>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>>>>>
>>>>>>>> I agree that we should be closing the connection
>>>>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> is an example of where we handled that in AS6). Probably after we
>>>>>>>> are
>>>>>>>> done with the MBeanServerConnection.
>>>>>>>>
>>>>>>>> I created ARQ-632 for this.
>>>>>>>>
>>>>>>>> I was expecting the cause to be a regression but instead perhaps
>>>>>>>> its
>>>>>>>> the performance improvements (we are running unit tests quicker,
>>>>>>>> causing more "jmx server connection timeout" threads to be
>>>>>>>> created).
>>>>>>>> Or, due to the number of tests in the testsuite now.
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "Brian Stansberry"<brian.stansberry at redhat.com>
>>>>>>>> To: jboss-as7-dev at lists.jboss.org
>>>>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>>>>>> synced with master yesterday
>>>>>>>>
>>>>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>>>>>> anything closing it. I'm not sure, but perhaps one of those is
>>>>>>>> created
>>>>>>>> per test method execution?
>>>>>>>>
>>>>>>>> I believe the "JMX server connection timeout 552" threads are
>>>>>>>> created by
>>>>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can
>>>>>>>> see the
>>>>>>>> relevant code in the stack trace. A quick look at that code leads
>>>>>>>> me to
>>>>>>>> believe those threads will by default live for 120 secs after the
>>>>>>>> last
>>>>>>>> request on the connection.
>>>>>>>>
>>>>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>>>>>> 1.6.0_24-b07.
>>>>>>>>>>>
>>>>>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>>>>>
>>>>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try
>>>>>>>>>>> that
>>>>>>>>>>> or we could start comparing environment settings.
>>>>>>>>>>>
>>>>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>>>>>
>>>>>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>>>>>
>>>>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>>>>>
>>>>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that
>>>>>>>>>> makes
>>>>>>>>>> a diff.
>>>>>>>>>
>>>>>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>>>>>> jdk1.6.0_24).
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Scott
>>>>>>>>>>>
>>>>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>>>>>>> but the
>>>>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory :
>>>>>>>>>>>> Cannot
>>>>>>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>>>>>>> Hudson
>>>>>>>>>>>> instance, don't remember). But I haven't seen them after
>>>>>>>>>>>> that. It
>>>>>>>>>>>> might
>>>>>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>>>>>> running
>>>>>>>>>>>> into this always?
>>>>>>>>>>>>
>>>>>>>>>>>> -Jaikiran
>>>>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>>>>>> How did these testsuite failures slip in? From my own
>>>>>>>>>>>>> experience
>>>>>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>>>>>> allTests
>>>>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>>>>>> merged some
>>>>>>>>>>>>> changes and only ran the allTests profile. I think that we
>>>>>>>>>>>>> need to
>>>>>>>>>>>>> continue to use -DallTests instead (that includes the
>>>>>>>>>>>>> integration
>>>>>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>>>>>>> seems we
>>>>>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>>>>>> rollback
>>>>>>>>>>>>> some changes until it goes away.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>>>>>> up. I can
>>>>>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Scott
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>>>>>> passing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When you say that your using a fresh master, is that a new
>>>>>>>>>>>>>> AS7
>>>>>>>>>>>>>> source
>>>>>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> server.log:
>>>>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Which folder are you starting the unit test from and what is
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>> command line?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>>>>>> (was using
>>>>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone
>>>>>>>>>>>>>>> changed
>>>>>>>>>>>>>>> that but
>>>>>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>>>>>> caused by
>>>>>>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>>>>>>> deploy
>>>>>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ondra
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Scott
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> jboss-as7-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>
>

-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat