[jboss-as7-dev] ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

Scott Marlow smarlow at redhat.com
Thu Oct 20 20:13:19 EDT 2011 

On 10/20/2011 05:42 PM, Brian Stansberry wrote:
> Can you try this:
>
> https://github.com/bstansberry/jboss-as/commit/a3dc2b2730c6c0aedf36257470d0ea8281802088

I updated my local copy of your branch and gave it a whirl.  The 
integration tests passed!  Nice patch!  :)


>
>
> It's the right thing to do anyway, as it should improve testsuite perf a
> fair bit by not opening connections all the time.
>
> On 10/20/11 3:43 PM, Brian Stansberry wrote:
>> I think the AS/ARQ integration (ManagementClient class) can easily
>> enough cache and re-use the JMXContext and thus the
>> MBeanServerConnection+JMXConnector. I'll send you a patch that does that
>> in a bit.
>>
>> On 10/20/11 3:22 PM, Scott Marlow wrote:
>>> It probably wasn't using the timeout setting added to
>>> as7/testsuite/integration/pom.xml. I changed the
>>> o.j.a.j.JMXConnectorService default to ten seconds and made it through
>>> the integration tests.
>>>
>>> If we change the default timeout to ten seconds (for a few days), that
>>> might cause test failures on busy/slower machines.
>>>
>>> On 10/20/2011 03:21 PM, Scott Marlow wrote:
>>>> No, I still see "java.lang.OutOfMemoryError: unable to create new
>>>> native
>>>> thread". I'll try a shorter timeout, maybe ten seconds instead of
>>>> thirty.
>>>>
>>>> Another workaround would be reducing the number of tests in
>>>> testsuite/integration (at least until ARQ-632 is fixed).
>>>>
>>>> On 10/20/2011 02:58 PM, Scott Marlow wrote:
>>>>> I'll give that a try.
>>>>>
>>>>> On 10/20/2011 02:36 PM, Brian Stansberry wrote:
>>>>>> I'm not seeing this failure, so it's hard to test a workaround.
>>>>>> But if
>>>>>> someone who is seeing it can give
>>>>>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
>>>>>>
>>>>>>
>>>>>>
>>>>>> a spin and let me know, I'd appreciate it.
>>>>>>
>>>>>> Probably easiest is just to check out and build
>>>>>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>>>>>>
>>>>>> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>>>>>>> See [1] for a good discussion of this; last post points at a
>>>>>>> possible
>>>>>>> workaround.
>>>>>>>
>>>>>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>>>>>>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>>>>>>> It is probably to do with the number of tests in the test suite, as
>>>>>>>> now all the tests that were previously in preview and spec are
>>>>>>>> executed in one run.
>>>>>>>>
>>>>>>>> Stuart
>>>>>>>>
>>>>>>>>
>>>>>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>>>>>>
>>>>>>>>> I agree that we should be closing the connection
>>>>>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> is an example of where we handled that in AS6). Probably after we
>>>>>>>>> are
>>>>>>>>> done with the MBeanServerConnection.
>>>>>>>>>
>>>>>>>>> I created ARQ-632 for this.
>>>>>>>>>
>>>>>>>>> I was expecting the cause to be a regression but instead perhaps
>>>>>>>>> its
>>>>>>>>> the performance improvements (we are running unit tests quicker,
>>>>>>>>> causing more "jmx server connection timeout" threads to be
>>>>>>>>> created).
>>>>>>>>> Or, due to the number of tests in the testsuite now.
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Brian Stansberry"<brian.stansberry at redhat.com>
>>>>>>>>> To: jboss-as7-dev at lists.jboss.org
>>>>>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>>>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>>>>>>> synced with master yesterday
>>>>>>>>>
>>>>>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>>>>>>> anything closing it. I'm not sure, but perhaps one of those is
>>>>>>>>> created
>>>>>>>>> per test method execution?
>>>>>>>>>
>>>>>>>>> I believe the "JMX server connection timeout 552" threads are
>>>>>>>>> created by
>>>>>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can
>>>>>>>>> see the
>>>>>>>>> relevant code in the stack trace. A quick look at that code leads
>>>>>>>>> me to
>>>>>>>>> believe those threads will by default live for 120 secs after the
>>>>>>>>> last
>>>>>>>>> request on the connection.
>>>>>>>>>
>>>>>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>>>>>>> 1.6.0_24-b07.
>>>>>>>>>>>>
>>>>>>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>>>>>>
>>>>>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could
>>>>>>>>>>>> try
>>>>>>>>>>>> that
>>>>>>>>>>>> or we could start comparing environment settings.
>>>>>>>>>>>>
>>>>>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>>>>>>
>>>>>>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>>>>>>
>>>>>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>>>>>>
>>>>>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that
>>>>>>>>>>> makes
>>>>>>>>>>> a diff.
>>>>>>>>>>
>>>>>>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>>>>>>> jdk1.6.0_24).
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Scott
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>>>>>>> I don't know about the test issues that Ondrej is running
>>>>>>>>>>>>> into,
>>>>>>>>>>>>> but the
>>>>>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory :
>>>>>>>>>>>>> Cannot
>>>>>>>>>>>>> create native thread) were encountered once by Carlo (or on
>>>>>>>>>>>>> our
>>>>>>>>>>>>> Hudson
>>>>>>>>>>>>> instance, don't remember). But I haven't seen them after
>>>>>>>>>>>>> that. It
>>>>>>>>>>>>> might
>>>>>>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>>>>>>> running
>>>>>>>>>>>>> into this always?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Jaikiran
>>>>>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>>>>>>> How did these testsuite failures slip in? From my own
>>>>>>>>>>>>>> experience
>>>>>>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>>>>>>> allTests
>>>>>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>>>>>>> merged some
>>>>>>>>>>>>>> changes and only ran the allTests profile. I think that we
>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>> continue to use -DallTests instead (that includes the
>>>>>>>>>>>>>> integration
>>>>>>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure why some people aren't seeing the failures
>>>>>>>>>>>>>> but it
>>>>>>>>>>>>>> seems we
>>>>>>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>>>>>>> rollback
>>>>>>>>>>>>>> some changes until it goes away.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>>>>>>> up. I can
>>>>>>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scott
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>>>>>>> passing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When you say that your using a fresh master, is that a new
>>>>>>>>>>>>>>> AS7
>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> server.log:
>>>>>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Which folder are you starting the unit test from and
>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>> command line?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>>>>>>> (was using
>>>>>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone
>>>>>>>>>>>>>>>> changed
>>>>>>>>>>>>>>>> that but
>>>>>>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>>>>>>> caused by
>>>>>>>>>>>>>>>>> improper use of deployer API, but giving it some time
>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>> deploy
>>>>>>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ondra
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and
>>>>>>>>>>>>>>>>>> do a
>>>>>>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Scott
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> jboss-as7-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> jboss-as7-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>
>>
>
>

More information about the jboss-as7-dev mailing list