[jboss-as7-dev] consistent testsuite failures since I synced with master yesterday

Thu Oct 20 14:36:38 EDT 2011

I'm not seeing this failure, so it's hard to test a workaround. But if 
someone who is seeing it can give 
https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333 
a spin and let me know, I'd appreciate it.

Probably easiest is just to check out and build 
https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak

On 10/20/11 11:49 AM, Brian Stansberry wrote:
> See [1] for a good discussion of this; last post points at a possible
> workaround.
>
> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>> It is probably to do with the number of tests in the test suite, as
>> now all the tests that were previously in preview and spec are
>> executed in one run.
>>
>> Stuart
>>
>>
>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>
>>> I agree that we should be closing the connection
>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>> is an example of where we handled that in AS6). Probably after we are
>>> done with the MBeanServerConnection.
>>>
>>> I created ARQ-632 for this.
>>>
>>> I was expecting the cause to be a regression but instead perhaps its
>>> the performance improvements (we are running unit tests quicker,
>>> causing more "jmx server connection timeout" threads to be created).
>>> Or, due to the number of tests in the testsuite now.
>>>
>>> ----- Original Message -----
>>> From: "Brian Stansberry"<brian.stansberry at redhat.com>
>>> To: jboss-as7-dev at lists.jboss.org
>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>> synced with master yesterday
>>>
>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>
>>> is creating a javax.management.remote.JMXConnector but I don't see
>>> anything closing it. I'm not sure, but perhaps one of those is created
>>> per test method execution?
>>>
>>> I believe the "JMX server connection timeout 552" threads are created by
>>> the JDK stuff AS7 is using for its JMX connector impl. You can see the
>>> relevant code in the stack trace. A quick look at that code leads me to
>>> believe those threads will by default live for 120 secs after the last
>>> request on the connection.
>>>
>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>> 1.6.0_24-b07.
>>>>>>
>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>
>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try that
>>>>>> or we could start comparing environment settings.
>>>>>>
>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>> -XX:MaxPermSize=128m"
>>>>>>
>>>>>> I don't have that set locally. I'll try that next...
>>>>>
>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>
>>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes
>>>>> a diff.
>>>>
>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>> jdk1.6.0_24).
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Scott
>>>>>>
>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>> but the
>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>> Hudson
>>>>>>> instance, don't remember). But I haven't seen them after that. It
>>>>>>> might
>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>> running
>>>>>>> into this always?
>>>>>>>
>>>>>>> -Jaikiran
>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>> How did these testsuite failures slip in? From my own experience
>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>> allTests
>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>> merged some
>>>>>>>> changes and only ran the allTests profile. I think that we need to
>>>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>>>> tests). Does anyone know more?
>>>>>>>>
>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>> seems we
>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>> rollback
>>>>>>>> some changes until it goes away.
>>>>>>>>
>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>> up. I can
>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>
>>>>>>>> Scott
>>>>>>>>
>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>> passing.
>>>>>>>>>
>>>>>>>>> When you say that your using a fresh master, is that a new AS7
>>>>>>>>> source
>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>
>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>
>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> server.log:
>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>
>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>
>>>>>>>>>> Which folder are you starting the unit test from and what is your
>>>>>>>>>> command line?
>>>>>>>>>>
>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>
>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>> (was using
>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed
>>>>>>>>>> that but
>>>>>>>>>> the test is still in queue.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>> caused by
>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>> deploy
>>>>>>>>>>> didn't help.
>>>>>>>>>>>
>>>>>>>>>>> Ondra
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>
>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>> fine for a
>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>> native thread"
>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>
>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>
>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>
>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>
>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>
>>>>>>>>>>>> Scott

-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat