[jboss-as7-dev] ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

Scott Marlow smarlow at redhat.com
Thu Oct 20 15:21:47 EDT 2011 

No, I still see "java.lang.OutOfMemoryError: unable to create new native 
thread".  I'll try a shorter timeout, maybe ten seconds instead of thirty.

Another workaround would be reducing the number of tests in 
testsuite/integration (at least until ARQ-632 is fixed).

On 10/20/2011 02:58 PM, Scott Marlow wrote:
> I'll give that a try.
>
> On 10/20/2011 02:36 PM, Brian Stansberry wrote:
>> I'm not seeing this failure, so it's hard to test a workaround. But if
>> someone who is seeing it can give
>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
>> a spin and let me know, I'd appreciate it.
>>
>> Probably easiest is just to check out and build
>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>>
>> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>>> See [1] for a good discussion of this; last post points at a possible
>>> workaround.
>>>
>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>>> It is probably to do with the number of tests in the test suite, as
>>>> now all the tests that were previously in preview and spec are
>>>> executed in one run.
>>>>
>>>> Stuart
>>>>
>>>>
>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>>
>>>>> I agree that we should be closing the connection
>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>>
>>>>> is an example of where we handled that in AS6). Probably after we are
>>>>> done with the MBeanServerConnection.
>>>>>
>>>>> I created ARQ-632 for this.
>>>>>
>>>>> I was expecting the cause to be a regression but instead perhaps its
>>>>> the performance improvements (we are running unit tests quicker,
>>>>> causing more "jmx server connection timeout" threads to be created).
>>>>> Or, due to the number of tests in the testsuite now.
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Brian Stansberry"<brian.stansberry at redhat.com>
>>>>> To: jboss-as7-dev at lists.jboss.org
>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>>> synced with master yesterday
>>>>>
>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>>
>>>>>
>>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>>> anything closing it. I'm not sure, but perhaps one of those is created
>>>>> per test method execution?
>>>>>
>>>>> I believe the "JMX server connection timeout 552" threads are
>>>>> created by
>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can see the
>>>>> relevant code in the stack trace. A quick look at that code leads me to
>>>>> believe those threads will by default live for 120 secs after the last
>>>>> request on the connection.
>>>>>
>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>>> 1.6.0_24-b07.
>>>>>>>>
>>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>>
>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try
>>>>>>>> that
>>>>>>>> or we could start comparing environment settings.
>>>>>>>>
>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>>
>>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>>
>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>>
>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes
>>>>>>> a diff.
>>>>>>
>>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>>> jdk1.6.0_24).
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Scott
>>>>>>>>
>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>>>> but the
>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>>>> Hudson
>>>>>>>>> instance, don't remember). But I haven't seen them after that. It
>>>>>>>>> might
>>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>>> running
>>>>>>>>> into this always?
>>>>>>>>>
>>>>>>>>> -Jaikiran
>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>>> How did these testsuite failures slip in? From my own experience
>>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>>> allTests
>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>>> merged some
>>>>>>>>>> changes and only ran the allTests profile. I think that we need to
>>>>>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>>
>>>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>>>> seems we
>>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>>> rollback
>>>>>>>>>> some changes until it goes away.
>>>>>>>>>>
>>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>>> up. I can
>>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>>
>>>>>>>>>> Scott
>>>>>>>>>>
>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>>> passing.
>>>>>>>>>>>
>>>>>>>>>>> When you say that your using a fresh master, is that a new AS7
>>>>>>>>>>> source
>>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> server.log:
>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>>
>>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>>
>>>>>>>>>>>> Which folder are you starting the unit test from and what is
>>>>>>>>>>>> your
>>>>>>>>>>>> command line?
>>>>>>>>>>>>
>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>>
>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>>> (was using
>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed
>>>>>>>>>>>> that but
>>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>>> caused by
>>>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>>>> deploy
>>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ondra
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scott
>>
>>
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

More information about the jboss-as7-dev mailing list