When running the as7/testsuite/integration tests, we run fine for a while but then "OutOfMemoryError: unable to create new native thread" errors start showing up in the server.log. Failing tests are here http://pastebin.com/D3SeJ2rP AS7 thread dump showing a lot of "JMX server connection timeout NNN" threads is here http://pastebin.com/qDC52WJG To recreate change into as7/testsuite/integration and do a "mvn clean install -DallTests". Is anyone not seeing this after syncing with master? Scott _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

I have a fresh master and testsuite runs fine. Except 2 failures (for which I also seek comments): org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase server.log: http://pastebin.test.redhat.com/64949

Anyone seeing that? I thought it could be a race condition caused by improper use of deployer API, but giving it some time after deploy didn't help. Ondra Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: > When running the as7/testsuite/integration tests, we run fine for a > while but then "OutOfMemoryError: unable to create new native thread" > errors start showing up in the server.log. > > Failing tests are here http://pastebin.com/D3SeJ2rP > > AS7 thread dump showing a lot of "JMX server connection timeout NNN" > threads is here http://pastebin.com/qDC52WJG > > To recreate change into as7/testsuite/integration and do a "mvn clean > install -DallTests". > > Is anyone not seeing this after syncing with master? > > Scott > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

The hudson AS7 test just finished with all integration tests passing. When you say that your using a fresh master, is that a new AS7 source tree or an existing one freshly synced with AS7 master? On 10/18/2011 12:26 PM, Scott Marlow wrote: > On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >> I have a fresh master and testsuite runs fine. >> Except 2 failures (for which I also seek comments): >> >> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >> >> server.log: >> http://pastebin.test.redhat.com/64949 > > I didn't see that failure. > > Which folder are you starting the unit test from and what is your > command line? > > I was trying to use "mvn clean install -DallTests" in the > as7/testsuite/integration folder. > > When I checked earlier, Hudson wasn't running the full test (was using > -PallTests instead of -DallTests). Looks like someone changed that but > the test is still in queue. > > >> >> Anyone seeing that? I thought it could be a race condition caused by >> improper use of deployer API, but giving it some time after deploy >> didn't help. >> >> Ondra >> >> >> >> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >> >>> When running the as7/testsuite/integration tests, we run fine for a >>> while but then "OutOfMemoryError: unable to create new native thread" >>> errors start showing up in the server.log. >>> >>> Failing tests are here http://pastebin.com/D3SeJ2rP >>> >>> AS7 thread dump showing a lot of "JMX server connection timeout NNN" >>> threads is here http://pastebin.com/qDC52WJG >>> >>> To recreate change into as7/testsuite/integration and do a "mvn clean >>> install -DallTests". >>> >>> Is anyone not seeing this after syncing with master? >>> >>> Scott >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

I don't know about the test issues that Ondrej is running into, but the ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot create native thread) were encountered once by Carlo (or on our Hudson instance, don't remember). But I haven't seen them after that. It might be environment specific. What JDK vendor/version? And are you running into this always? -Jaikiran On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: > How did these testsuite failures slip in? From my own experience > running the updated testsuite, it looks like we now have an allTests > profile that doesn't run the integration tests. Perhaps we merged some > changes and only ran the allTests profile. I think that we need to > continue to use -DallTests instead (that includes the integration > tests). Does anyone know more? > > I'm not sure why some people aren't seeing the failures but it seems we > need to track backwards from the symptoms (listed below) or rollback > some changes until it goes away. > > If anyone is working on tracking the cause down, please speak up. I can > jump in but don't want to duplicate effort. > > Scott > > On 10/18/2011 02:15 PM, Scott Marlow wrote: >> The hudson AS7 test just finished with all integration tests passing. >> >> When you say that your using a fresh master, is that a new AS7 source >> tree or an existing one freshly synced with AS7 master? >> >> >> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>> I have a fresh master and testsuite runs fine. >>>> Except 2 failures (for which I also seek comments): >>>> >>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>> >>>> server.log: >>>> http://pastebin.test.redhat.com/64949 >>> >>> I didn't see that failure. >>> >>> Which folder are you starting the unit test from and what is your >>> command line? >>> >>> I was trying to use "mvn clean install -DallTests" in the >>> as7/testsuite/integration folder. >>> >>> When I checked earlier, Hudson wasn't running the full test (was using >>> -PallTests instead of -DallTests). Looks like someone changed that but >>> the test is still in queue. >>> >>> >>>> >>>> Anyone seeing that? I thought it could be a race condition caused by >>>> improper use of deployer API, but giving it some time after deploy >>>> didn't help. >>>> >>>> Ondra >>>> >>>> >>>> >>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>> >>>>> When running the as7/testsuite/integration tests, we run fine for a >>>>> while but then "OutOfMemoryError: unable to create new native thread" >>>>> errors start showing up in the server.log. >>>>> >>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>> >>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN" >>>>> threads is here http://pastebin.com/qDC52WJG >>>>> >>>>> To recreate change into as7/testsuite/integration and do a "mvn clean >>>>> install -DallTests". >>>>> >>>>> Is anyone not seeing this after syncing with master? >>>>> >>>>> Scott >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

On 10/19/2011 11:01 AM, Scott Marlow wrote: > I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun > 1.6.0_24-b07. > > Jesper also got the failures with jdk 6 u26. > > I haven't tried running with my old jdk1.6.0_24 yet. I could try that > or we could start comparing environment settings. > > On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m > -XX:MaxPermSize=128m" > > I don't have that set locally. I'll try that next... Testsuite still fails for me with the above MAVEN_OPTS set. I'll my older jdk 6 (same as hudson is using) and see if that makes a diff.

> > Scott > > On 10/19/2011 10:49 AM, Jaikiran Pai wrote: >> I don't know about the test issues that Ondrej is running into, but the >> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot >> create native thread) were encountered once by Carlo (or on our Hudson >> instance, don't remember). But I haven't seen them after that. It might >> be environment specific. What JDK vendor/version? And are you running >> into this always? >> >> -Jaikiran >> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: >>> How did these testsuite failures slip in? From my own experience >>> running the updated testsuite, it looks like we now have an allTests >>> profile that doesn't run the integration tests. Perhaps we merged some >>> changes and only ran the allTests profile. I think that we need to >>> continue to use -DallTests instead (that includes the integration >>> tests). Does anyone know more? >>> >>> I'm not sure why some people aren't seeing the failures but it seems we >>> need to track backwards from the symptoms (listed below) or rollback >>> some changes until it goes away. >>> >>> If anyone is working on tracking the cause down, please speak up. I can >>> jump in but don't want to duplicate effort. >>> >>> Scott >>> >>> On 10/18/2011 02:15 PM, Scott Marlow wrote: >>>> The hudson AS7 test just finished with all integration tests passing. >>>> >>>> When you say that your using a fresh master, is that a new AS7 source >>>> tree or an existing one freshly synced with AS7 master? >>>> >>>> >>>> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>>>> I have a fresh master and testsuite runs fine. >>>>>> Except 2 failures (for which I also seek comments): >>>>>> >>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>>>> >>>>>> server.log: >>>>>> http://pastebin.test.redhat.com/64949 >>>>> >>>>> I didn't see that failure. >>>>> >>>>> Which folder are you starting the unit test from and what is your >>>>> command line? >>>>> >>>>> I was trying to use "mvn clean install -DallTests" in the >>>>> as7/testsuite/integration folder. >>>>> >>>>> When I checked earlier, Hudson wasn't running the full test (was using >>>>> -PallTests instead of -DallTests). Looks like someone changed that but >>>>> the test is still in queue. >>>>> >>>>> >>>>>> >>>>>> Anyone seeing that? I thought it could be a race condition caused by >>>>>> improper use of deployer API, but giving it some time after deploy >>>>>> didn't help. >>>>>> >>>>>> Ondra >>>>>> >>>>>> >>>>>> >>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>>>> >>>>>>> When running the as7/testsuite/integration tests, we run fine for a >>>>>>> while but then "OutOfMemoryError: unable to create new native thread" >>>>>>> errors start showing up in the server.log. >>>>>>> >>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>>>> >>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN" >>>>>>> threads is here http://pastebin.com/qDC52WJG >>>>>>> >>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean >>>>>>> install -DallTests". >>>>>>> >>>>>>> Is anyone not seeing this after syncing with master? >>>>>>> >>>>>>> Scott >>>>>>> _______________________________________________ >>>>>>> jboss-as7-dev mailing list >>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>> >>>>> >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

On 10/19/2011 12:13 PM, Scott Marlow wrote: > On 10/19/2011 11:01 AM, Scott Marlow wrote: >> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun >> 1.6.0_24-b07. >> >> Jesper also got the failures with jdk 6 u26. >> >> I haven't tried running with my old jdk1.6.0_24 yet. I could try that >> or we could start comparing environment settings. >> >> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m >> -XX:MaxPermSize=128m" >> >> I don't have that set locally. I'll try that next... > > Testsuite still fails for me with the above MAVEN_OPTS set. > > I'll my older jdk 6 (same as hudson is using) and see if that makes a diff. Same failure still (with MAVEN_OPTS set as above and using jdk1.6.0_24). > > >> >> Scott >> >> On 10/19/2011 10:49 AM, Jaikiran Pai wrote: >>> I don't know about the test issues that Ondrej is running into, but the >>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot >>> create native thread) were encountered once by Carlo (or on our Hudson >>> instance, don't remember). But I haven't seen them after that. It might >>> be environment specific. What JDK vendor/version? And are you running >>> into this always? >>> >>> -Jaikiran >>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: >>>> How did these testsuite failures slip in? From my own experience >>>> running the updated testsuite, it looks like we now have an allTests >>>> profile that doesn't run the integration tests. Perhaps we merged some >>>> changes and only ran the allTests profile. I think that we need to >>>> continue to use -DallTests instead (that includes the integration >>>> tests). Does anyone know more? >>>> >>>> I'm not sure why some people aren't seeing the failures but it seems we >>>> need to track backwards from the symptoms (listed below) or rollback >>>> some changes until it goes away. >>>> >>>> If anyone is working on tracking the cause down, please speak up. I can >>>> jump in but don't want to duplicate effort. >>>> >>>> Scott >>>> >>>> On 10/18/2011 02:15 PM, Scott Marlow wrote: >>>>> The hudson AS7 test just finished with all integration tests passing. >>>>> >>>>> When you say that your using a fresh master, is that a new AS7 source >>>>> tree or an existing one freshly synced with AS7 master? >>>>> >>>>> >>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>>>>> I have a fresh master and testsuite runs fine. >>>>>>> Except 2 failures (for which I also seek comments): >>>>>>> >>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>>>>> >>>>>>> server.log: >>>>>>> http://pastebin.test.redhat.com/64949 >>>>>> >>>>>> I didn't see that failure. >>>>>> >>>>>> Which folder are you starting the unit test from and what is your >>>>>> command line? >>>>>> >>>>>> I was trying to use "mvn clean install -DallTests" in the >>>>>> as7/testsuite/integration folder. >>>>>> >>>>>> When I checked earlier, Hudson wasn't running the full test (was using >>>>>> -PallTests instead of -DallTests). Looks like someone changed that but >>>>>> the test is still in queue. >>>>>> >>>>>> >>>>>>> >>>>>>> Anyone seeing that? I thought it could be a race condition caused by >>>>>>> improper use of deployer API, but giving it some time after deploy >>>>>>> didn't help. >>>>>>> >>>>>>> Ondra >>>>>>> >>>>>>> >>>>>>> >>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>>>>> >>>>>>>> When running the as7/testsuite/integration tests, we run fine for a >>>>>>>> while but then "OutOfMemoryError: unable to create new native thread" >>>>>>>> errors start showing up in the server.log. >>>>>>>> >>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>>>>> >>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN" >>>>>>>> threads is here http://pastebin.com/qDC52WJG >>>>>>>> >>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean >>>>>>>> install -DallTests". >>>>>>>> >>>>>>>> Is anyone not seeing this after syncing with master? >>>>>>>> >>>>>>>> Scott >>>>>>>> _______________________________________________ >>>>>>>> jboss-as7-dev mailing list >>>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> jboss-as7-dev mailing list >>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>> >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

It is probably to do with the number of tests in the test suite, as now all the tests that were previously in preview and spec are executed in one run. Stuart On 20/10/2011, at 10:24 AM, Scott Marlow wrote: > I agree that we should be closing the connection (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/j... is an example of where we handled that in AS6). Probably after we are done with the MBeanServerConnection. > > I created ARQ-632 for this. > > I was expecting the cause to be a regression but instead perhaps its the performance improvements (we are running unit tests quicker, causing more "jmx server connection timeout" threads to be created). Or, due to the number of tests in the testsuite now. > > ----- Original Message ----- > From: "Brian Stansberry"<brian.stansberry(a)redhat.com> > To: jboss-as7-dev(a)lists.jboss.org > Sent: Wednesday, October 19, 2011 5:47:41 PM > Subject: Re: [jboss-as7-dev] consistent testsuite failures since I synced with master yesterday > > org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection() > is creating a javax.management.remote.JMXConnector but I don't see > anything closing it. I'm not sure, but perhaps one of those is created > per test method execution? > > I believe the "JMX server connection timeout 552" threads are created by > the JDK stuff AS7 is using for its JMX connector impl. You can see the > relevant code in the stack trace. A quick look at that code leads me to > believe those threads will by default live for 120 secs after the last > request on the connection. > > On 10/19/11 11:54 AM, Scott Marlow wrote: >> On 10/19/2011 12:13 PM, Scott Marlow wrote: >>> On 10/19/2011 11:01 AM, Scott Marlow wrote: >>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun >>>> 1.6.0_24-b07. >>>> >>>> Jesper also got the failures with jdk 6 u26. >>>> >>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try that >>>> or we could start comparing environment settings. >>>> >>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m >>>> -XX:MaxPermSize=128m" >>>> >>>> I don't have that set locally. I'll try that next... >>> >>> Testsuite still fails for me with the above MAVEN_OPTS set. >>> >>> I'll my older jdk 6 (same as hudson is using) and see if that makes a diff. >> >> Same failure still (with MAVEN_OPTS set as above and using jdk1.6.0_24). >> >>> >>> >>>> >>>> Scott >>>> >>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote: >>>>> I don't know about the test issues that Ondrej is running into, but the >>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot >>>>> create native thread) were encountered once by Carlo (or on our Hudson >>>>> instance, don't remember). But I haven't seen them after that. It might >>>>> be environment specific. What JDK vendor/version? And are you running >>>>> into this always? >>>>> >>>>> -Jaikiran >>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: >>>>>> How did these testsuite failures slip in? From my own experience >>>>>> running the updated testsuite, it looks like we now have an allTests >>>>>> profile that doesn't run the integration tests. Perhaps we merged some >>>>>> changes and only ran the allTests profile. I think that we need to >>>>>> continue to use -DallTests instead (that includes the integration >>>>>> tests). Does anyone know more? >>>>>> >>>>>> I'm not sure why some people aren't seeing the failures but it seems we >>>>>> need to track backwards from the symptoms (listed below) or rollback >>>>>> some changes until it goes away. >>>>>> >>>>>> If anyone is working on tracking the cause down, please speak up. I can >>>>>> jump in but don't want to duplicate effort. >>>>>> >>>>>> Scott >>>>>> >>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote: >>>>>>> The hudson AS7 test just finished with all integration tests passing. >>>>>>> >>>>>>> When you say that your using a fresh master, is that a new AS7 source >>>>>>> tree or an existing one freshly synced with AS7 master? >>>>>>> >>>>>>> >>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>>>>>>> I have a fresh master and testsuite runs fine. >>>>>>>>> Except 2 failures (for which I also seek comments): >>>>>>>>> >>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>>>>>>> >>>>>>>>> server.log: >>>>>>>>> http://pastebin.test.redhat.com/64949 >>>>>>>> >>>>>>>> I didn't see that failure. >>>>>>>> >>>>>>>> Which folder are you starting the unit test from and what is your >>>>>>>> command line? >>>>>>>> >>>>>>>> I was trying to use "mvn clean install -DallTests" in the >>>>>>>> as7/testsuite/integration folder. >>>>>>>> >>>>>>>> When I checked earlier, Hudson wasn't running the full test (was using >>>>>>>> -PallTests instead of -DallTests). Looks like someone changed that but >>>>>>>> the test is still in queue. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Anyone seeing that? I thought it could be a race condition caused by >>>>>>>>> improper use of deployer API, but giving it some time after deploy >>>>>>>>> didn't help. >>>>>>>>> >>>>>>>>> Ondra >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>>>>>>> >>>>>>>>>> When running the as7/testsuite/integration tests, we run fine for a >>>>>>>>>> while but then "OutOfMemoryError: unable to create new native thread" >>>>>>>>>> errors start showing up in the server.log. >>>>>>>>>> >>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>>>>>>> >>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN" >>>>>>>>>> threads is here http://pastebin.com/qDC52WJG >>>>>>>>>> >>>>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean >>>>>>>>>> install -DallTests". >>>>>>>>>> >>>>>>>>>> Is anyone not seeing this after syncing with master? >>>>>>>>>> >>>>>>>>>> Scott >>>>>>>>>> _______________________________________________ >>>>>>>>>> jboss-as7-dev mailing list >>>>>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> jboss-as7-dev mailing list >>>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>>> >>>>>>> _______________________________________________ >>>>>>> jboss-as7-dev mailing list >>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>> >>>>>> _______________________________________________ >>>>>> jboss-as7-dev mailing list >>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>> >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > > -- > Brian Stansberry > Principal Software Engineer > JBoss by Red Hat > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

See [1] for a good discussion of this; last post points at a possible workaround. [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966 On 10/19/11 6:26 PM, Stuart Douglas wrote: > It is probably to do with the number of tests in the test suite, as > now all the tests that were previously in preview and spec are > executed in one run. > > Stuart > > > On 20/10/2011, at 10:24 AM, Scott Marlow wrote: > >> I agree that we should be closing the connection >> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/j... >> is an example of where we handled that in AS6). Probably after we are >> done with the MBeanServerConnection. >> >> I created ARQ-632 for this. >> >> I was expecting the cause to be a regression but instead perhaps its >> the performance improvements (we are running unit tests quicker, >> causing more "jmx server connection timeout" threads to be created). >> Or, due to the number of tests in the testsuite now. >> >> ----- Original Message ----- >> From: "Brian Stansberry"<brian.stansberry(a)redhat.com> >> To: jboss-as7-dev(a)lists.jboss.org >> Sent: Wednesday, October 19, 2011 5:47:41 PM >> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I >> synced with master yesterday >> >> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection() >> >> is creating a javax.management.remote.JMXConnector but I don't see >> anything closing it. I'm not sure, but perhaps one of those is created >> per test method execution? >> >> I believe the "JMX server connection timeout 552" threads are created by >> the JDK stuff AS7 is using for its JMX connector impl. You can see the >> relevant code in the stack trace. A quick look at that code leads me to >> believe those threads will by default live for 120 secs after the last >> request on the connection. >> >> On 10/19/11 11:54 AM, Scott Marlow wrote: >>> On 10/19/2011 12:13 PM, Scott Marlow wrote: >>>> On 10/19/2011 11:01 AM, Scott Marlow wrote: >>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun >>>>> 1.6.0_24-b07. >>>>> >>>>> Jesper also got the failures with jdk 6 u26. >>>>> >>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try that >>>>> or we could start comparing environment settings. >>>>> >>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m >>>>> -XX:MaxPermSize=128m" >>>>> >>>>> I don't have that set locally. I'll try that next... >>>> >>>> Testsuite still fails for me with the above MAVEN_OPTS set. >>>> >>>> I'll my older jdk 6 (same as hudson is using) and see if that makes >>>> a diff. >>> >>> Same failure still (with MAVEN_OPTS set as above and using >>> jdk1.6.0_24). >>> >>>> >>>> >>>>> >>>>> Scott >>>>> >>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote: >>>>>> I don't know about the test issues that Ondrej is running into, >>>>>> but the >>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot >>>>>> create native thread) were encountered once by Carlo (or on our >>>>>> Hudson >>>>>> instance, don't remember). But I haven't seen them after that. It >>>>>> might >>>>>> be environment specific. What JDK vendor/version? And are you >>>>>> running >>>>>> into this always? >>>>>> >>>>>> -Jaikiran >>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: >>>>>>> How did these testsuite failures slip in? From my own experience >>>>>>> running the updated testsuite, it looks like we now have an >>>>>>> allTests >>>>>>> profile that doesn't run the integration tests. Perhaps we >>>>>>> merged some >>>>>>> changes and only ran the allTests profile. I think that we need to >>>>>>> continue to use -DallTests instead (that includes the integration >>>>>>> tests). Does anyone know more? >>>>>>> >>>>>>> I'm not sure why some people aren't seeing the failures but it >>>>>>> seems we >>>>>>> need to track backwards from the symptoms (listed below) or >>>>>>> rollback >>>>>>> some changes until it goes away. >>>>>>> >>>>>>> If anyone is working on tracking the cause down, please speak >>>>>>> up. I can >>>>>>> jump in but don't want to duplicate effort. >>>>>>> >>>>>>> Scott >>>>>>> >>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote: >>>>>>>> The hudson AS7 test just finished with all integration tests >>>>>>>> passing. >>>>>>>> >>>>>>>> When you say that your using a fresh master, is that a new AS7 >>>>>>>> source >>>>>>>> tree or an existing one freshly synced with AS7 master? >>>>>>>> >>>>>>>> >>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>>>>>>>> I have a fresh master and testsuite runs fine. >>>>>>>>>> Except 2 failures (for which I also seek comments): >>>>>>>>>> >>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>>>>>>>> >>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> server.log: >>>>>>>>>> http://pastebin.test.redhat.com/64949 >>>>>>>>> >>>>>>>>> I didn't see that failure. >>>>>>>>> >>>>>>>>> Which folder are you starting the unit test from and what is your >>>>>>>>> command line? >>>>>>>>> >>>>>>>>> I was trying to use "mvn clean install -DallTests" in the >>>>>>>>> as7/testsuite/integration folder. >>>>>>>>> >>>>>>>>> When I checked earlier, Hudson wasn't running the full test >>>>>>>>> (was using >>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed >>>>>>>>> that but >>>>>>>>> the test is still in queue. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Anyone seeing that? I thought it could be a race condition >>>>>>>>>> caused by >>>>>>>>>> improper use of deployer API, but giving it some time after >>>>>>>>>> deploy >>>>>>>>>> didn't help. >>>>>>>>>> >>>>>>>>>> Ondra >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>>>>>>>> >>>>>>>>>>> When running the as7/testsuite/integration tests, we run >>>>>>>>>>> fine for a >>>>>>>>>>> while but then "OutOfMemoryError: unable to create new >>>>>>>>>>> native thread" >>>>>>>>>>> errors start showing up in the server.log. >>>>>>>>>>> >>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>>>>>>>> >>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection >>>>>>>>>>> timeout NNN" >>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG >>>>>>>>>>> >>>>>>>>>>> To recreate change into as7/testsuite/integration and do a >>>>>>>>>>> "mvn clean >>>>>>>>>>> install -DallTests". >>>>>>>>>>> >>>>>>>>>>> Is anyone not seeing this after syncing with master? >>>>>>>>>>> >>>>>>>>>>> Scott

I'll give that a try. On 10/20/2011 02:36 PM, Brian Stansberry wrote: > I'm not seeing this failure, so it's hard to test a workaround. But if > someone who is seeing it can give > https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139... > a spin and let me know, I'd appreciate it. > > Probably easiest is just to check out and build > https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak > > On 10/20/11 11:49 AM, Brian Stansberry wrote: >> See [1] for a good discussion of this; last post points at a possible >> workaround. >> >> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966 >> On 10/19/11 6:26 PM, Stuart Douglas wrote: >>> It is probably to do with the number of tests in the test suite, as >>> now all the tests that were previously in preview and spec are >>> executed in one run. >>> >>> Stuart >>> >>> >>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote: >>> >>>> I agree that we should be closing the connection >>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/j... >>>> >>>> is an example of where we handled that in AS6). Probably after we are >>>> done with the MBeanServerConnection. >>>> >>>> I created ARQ-632 for this. >>>> >>>> I was expecting the cause to be a regression but instead perhaps its >>>> the performance improvements (we are running unit tests quicker, >>>> causing more "jmx server connection timeout" threads to be created). >>>> Or, due to the number of tests in the testsuite now. >>>> >>>> ----- Original Message ----- >>>> From: "Brian Stansberry"<brian.stansberry(a)redhat.com> >>>> To: jboss-as7-dev(a)lists.jboss.org >>>> Sent: Wednesday, October 19, 2011 5:47:41 PM >>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I >>>> synced with master yesterday >>>> >>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection() >>>> >>>> >>>> is creating a javax.management.remote.JMXConnector but I don't see >>>> anything closing it. I'm not sure, but perhaps one of those is created >>>> per test method execution? >>>> >>>> I believe the "JMX server connection timeout 552" threads are >>>> created by >>>> the JDK stuff AS7 is using for its JMX connector impl. You can see the >>>> relevant code in the stack trace. A quick look at that code leads me to >>>> believe those threads will by default live for 120 secs after the last >>>> request on the connection. >>>> >>>> On 10/19/11 11:54 AM, Scott Marlow wrote: >>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote: >>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote: >>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun >>>>>>> 1.6.0_24-b07. >>>>>>> >>>>>>> Jesper also got the failures with jdk 6 u26. >>>>>>> >>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try >>>>>>> that >>>>>>> or we could start comparing environment settings. >>>>>>> >>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m >>>>>>> -XX:MaxPermSize=128m" >>>>>>> >>>>>>> I don't have that set locally. I'll try that next... >>>>>> >>>>>> Testsuite still fails for me with the above MAVEN_OPTS set. >>>>>> >>>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes >>>>>> a diff. >>>>> >>>>> Same failure still (with MAVEN_OPTS set as above and using >>>>> jdk1.6.0_24). >>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Scott >>>>>>> >>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote: >>>>>>>> I don't know about the test issues that Ondrej is running into, >>>>>>>> but the >>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot >>>>>>>> create native thread) were encountered once by Carlo (or on our >>>>>>>> Hudson >>>>>>>> instance, don't remember). But I haven't seen them after that. It >>>>>>>> might >>>>>>>> be environment specific. What JDK vendor/version? And are you >>>>>>>> running >>>>>>>> into this always? >>>>>>>> >>>>>>>> -Jaikiran >>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: >>>>>>>>> How did these testsuite failures slip in? From my own experience >>>>>>>>> running the updated testsuite, it looks like we now have an >>>>>>>>> allTests >>>>>>>>> profile that doesn't run the integration tests. Perhaps we >>>>>>>>> merged some >>>>>>>>> changes and only ran the allTests profile. I think that we need to >>>>>>>>> continue to use -DallTests instead (that includes the integration >>>>>>>>> tests). Does anyone know more? >>>>>>>>> >>>>>>>>> I'm not sure why some people aren't seeing the failures but it >>>>>>>>> seems we >>>>>>>>> need to track backwards from the symptoms (listed below) or >>>>>>>>> rollback >>>>>>>>> some changes until it goes away. >>>>>>>>> >>>>>>>>> If anyone is working on tracking the cause down, please speak >>>>>>>>> up. I can >>>>>>>>> jump in but don't want to duplicate effort. >>>>>>>>> >>>>>>>>> Scott >>>>>>>>> >>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote: >>>>>>>>>> The hudson AS7 test just finished with all integration tests >>>>>>>>>> passing. >>>>>>>>>> >>>>>>>>>> When you say that your using a fresh master, is that a new AS7 >>>>>>>>>> source >>>>>>>>>> tree or an existing one freshly synced with AS7 master? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>>>>>>>>>> I have a fresh master and testsuite runs fine. >>>>>>>>>>>> Except 2 failures (for which I also seek comments): >>>>>>>>>>>> >>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> server.log: >>>>>>>>>>>> http://pastebin.test.redhat.com/64949 >>>>>>>>>>> >>>>>>>>>>> I didn't see that failure. >>>>>>>>>>> >>>>>>>>>>> Which folder are you starting the unit test from and what is >>>>>>>>>>> your >>>>>>>>>>> command line? >>>>>>>>>>> >>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the >>>>>>>>>>> as7/testsuite/integration folder. >>>>>>>>>>> >>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test >>>>>>>>>>> (was using >>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed >>>>>>>>>>> that but >>>>>>>>>>> the test is still in queue. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition >>>>>>>>>>>> caused by >>>>>>>>>>>> improper use of deployer API, but giving it some time after >>>>>>>>>>>> deploy >>>>>>>>>>>> didn't help. >>>>>>>>>>>> >>>>>>>>>>>> Ondra >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>>>>>>>>>> >>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run >>>>>>>>>>>>> fine for a >>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new >>>>>>>>>>>>> native thread" >>>>>>>>>>>>> errors start showing up in the server.log. >>>>>>>>>>>>> >>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>>>>>>>>>> >>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection >>>>>>>>>>>>> timeout NNN" >>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG >>>>>>>>>>>>> >>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a >>>>>>>>>>>>> "mvn clean >>>>>>>>>>>>> install -DallTests". >>>>>>>>>>>>> >>>>>>>>>>>>> Is anyone not seeing this after syncing with master? >>>>>>>>>>>>> >>>>>>>>>>>>> Scott > > _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

It probably wasn't using the timeout setting added to as7/testsuite/integration/pom.xml. I changed the o.j.a.j.JMXConnectorService default to ten seconds and made it through the integration tests. If we change the default timeout to ten seconds (for a few days), that might cause test failures on busy/slower machines. On 10/20/2011 03:21 PM, Scott Marlow wrote: > No, I still see "java.lang.OutOfMemoryError: unable to create new native > thread". I'll try a shorter timeout, maybe ten seconds instead of thirty. > > Another workaround would be reducing the number of tests in > testsuite/integration (at least until ARQ-632 is fixed). > > On 10/20/2011 02:58 PM, Scott Marlow wrote: >> I'll give that a try. >> >> On 10/20/2011 02:36 PM, Brian Stansberry wrote: >>> I'm not seeing this failure, so it's hard to test a workaround. But if >>> someone who is seeing it can give >>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139... >>> >>> a spin and let me know, I'd appreciate it. >>> >>> Probably easiest is just to check out and build >>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak >>> >>> On 10/20/11 11:49 AM, Brian Stansberry wrote: >>>> See [1] for a good discussion of this; last post points at a possible >>>> workaround. >>>> >>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966 >>>> On 10/19/11 6:26 PM, Stuart Douglas wrote: >>>>> It is probably to do with the number of tests in the test suite, as >>>>> now all the tests that were previously in preview and spec are >>>>> executed in one run. >>>>> >>>>> Stuart >>>>> >>>>> >>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote: >>>>> >>>>>> I agree that we should be closing the connection >>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/j... >>>>>> >>>>>> >>>>>> is an example of where we handled that in AS6). Probably after we >>>>>> are >>>>>> done with the MBeanServerConnection. >>>>>> >>>>>> I created ARQ-632 for this. >>>>>> >>>>>> I was expecting the cause to be a regression but instead perhaps its >>>>>> the performance improvements (we are running unit tests quicker, >>>>>> causing more "jmx server connection timeout" threads to be created). >>>>>> Or, due to the number of tests in the testsuite now. >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Brian Stansberry"<brian.stansberry(a)redhat.com> >>>>>> To: jboss-as7-dev(a)lists.jboss.org >>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM >>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I >>>>>> synced with master yesterday >>>>>> >>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection() >>>>>> >>>>>> >>>>>> >>>>>> is creating a javax.management.remote.JMXConnector but I don't see >>>>>> anything closing it. I'm not sure, but perhaps one of those is >>>>>> created >>>>>> per test method execution? >>>>>> >>>>>> I believe the "JMX server connection timeout 552" threads are >>>>>> created by >>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can >>>>>> see the >>>>>> relevant code in the stack trace. A quick look at that code leads >>>>>> me to >>>>>> believe those threads will by default live for 120 secs after the >>>>>> last >>>>>> request on the connection. >>>>>> >>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote: >>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote: >>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote: >>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun >>>>>>>>> 1.6.0_24-b07. >>>>>>>>> >>>>>>>>> Jesper also got the failures with jdk 6 u26. >>>>>>>>> >>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try >>>>>>>>> that >>>>>>>>> or we could start comparing environment settings. >>>>>>>>> >>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m >>>>>>>>> -XX:MaxPermSize=128m" >>>>>>>>> >>>>>>>>> I don't have that set locally. I'll try that next... >>>>>>>> >>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set. >>>>>>>> >>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that >>>>>>>> makes >>>>>>>> a diff. >>>>>>> >>>>>>> Same failure still (with MAVEN_OPTS set as above and using >>>>>>> jdk1.6.0_24). >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Scott >>>>>>>>> >>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote: >>>>>>>>>> I don't know about the test issues that Ondrej is running into, >>>>>>>>>> but the >>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : >>>>>>>>>> Cannot >>>>>>>>>> create native thread) were encountered once by Carlo (or on our >>>>>>>>>> Hudson >>>>>>>>>> instance, don't remember). But I haven't seen them after >>>>>>>>>> that. It >>>>>>>>>> might >>>>>>>>>> be environment specific. What JDK vendor/version? And are you >>>>>>>>>> running >>>>>>>>>> into this always? >>>>>>>>>> >>>>>>>>>> -Jaikiran >>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: >>>>>>>>>>> How did these testsuite failures slip in? From my own >>>>>>>>>>> experience >>>>>>>>>>> running the updated testsuite, it looks like we now have an >>>>>>>>>>> allTests >>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we >>>>>>>>>>> merged some >>>>>>>>>>> changes and only ran the allTests profile. I think that we >>>>>>>>>>> need to >>>>>>>>>>> continue to use -DallTests instead (that includes the >>>>>>>>>>> integration >>>>>>>>>>> tests). Does anyone know more? >>>>>>>>>>> >>>>>>>>>>> I'm not sure why some people aren't seeing the failures but it >>>>>>>>>>> seems we >>>>>>>>>>> need to track backwards from the symptoms (listed below) or >>>>>>>>>>> rollback >>>>>>>>>>> some changes until it goes away. >>>>>>>>>>> >>>>>>>>>>> If anyone is working on tracking the cause down, please speak >>>>>>>>>>> up. I can >>>>>>>>>>> jump in but don't want to duplicate effort. >>>>>>>>>>> >>>>>>>>>>> Scott >>>>>>>>>>> >>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote: >>>>>>>>>>>> The hudson AS7 test just finished with all integration tests >>>>>>>>>>>> passing. >>>>>>>>>>>> >>>>>>>>>>>> When you say that your using a fresh master, is that a new AS7 >>>>>>>>>>>> source >>>>>>>>>>>> tree or an existing one freshly synced with AS7 master? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>>>>>>>>>>>> I have a fresh master and testsuite runs fine. >>>>>>>>>>>>>> Except 2 failures (for which I also seek comments): >>>>>>>>>>>>>> >>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> server.log: >>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949 >>>>>>>>>>>>> >>>>>>>>>>>>> I didn't see that failure. >>>>>>>>>>>>> >>>>>>>>>>>>> Which folder are you starting the unit test from and what is >>>>>>>>>>>>> your >>>>>>>>>>>>> command line? >>>>>>>>>>>>> >>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the >>>>>>>>>>>>> as7/testsuite/integration folder. >>>>>>>>>>>>> >>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test >>>>>>>>>>>>> (was using >>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed >>>>>>>>>>>>> that but >>>>>>>>>>>>> the test is still in queue. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition >>>>>>>>>>>>>> caused by >>>>>>>>>>>>>> improper use of deployer API, but giving it some time after >>>>>>>>>>>>>> deploy >>>>>>>>>>>>>> didn't help. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ondra >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run >>>>>>>>>>>>>>> fine for a >>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new >>>>>>>>>>>>>>> native thread" >>>>>>>>>>>>>>> errors start showing up in the server.log. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection >>>>>>>>>>>>>>> timeout NNN" >>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a >>>>>>>>>>>>>>> "mvn clean >>>>>>>>>>>>>>> install -DallTests". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Scott >>> >>> >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Can you try this: https://github.com/bstansberry/jboss-as/commit/a3dc2b2730c6c0aedf36257470...

It's the right thing to do anyway, as it should improve testsuite perf a fair bit by not opening connections all the time. On 10/20/11 3:43 PM, Brian Stansberry wrote: > I think the AS/ARQ integration (ManagementClient class) can easily > enough cache and re-use the JMXContext and thus the > MBeanServerConnection+JMXConnector. I'll send you a patch that does that > in a bit. > > On 10/20/11 3:22 PM, Scott Marlow wrote: >> It probably wasn't using the timeout setting added to >> as7/testsuite/integration/pom.xml. I changed the >> o.j.a.j.JMXConnectorService default to ten seconds and made it through >> the integration tests. >> >> If we change the default timeout to ten seconds (for a few days), that >> might cause test failures on busy/slower machines. >> >> On 10/20/2011 03:21 PM, Scott Marlow wrote: >>> No, I still see "java.lang.OutOfMemoryError: unable to create new >>> native >>> thread". I'll try a shorter timeout, maybe ten seconds instead of >>> thirty. >>> >>> Another workaround would be reducing the number of tests in >>> testsuite/integration (at least until ARQ-632 is fixed). >>> >>> On 10/20/2011 02:58 PM, Scott Marlow wrote: >>>> I'll give that a try. >>>> >>>> On 10/20/2011 02:36 PM, Brian Stansberry wrote: >>>>> I'm not seeing this failure, so it's hard to test a workaround. >>>>> But if >>>>> someone who is seeing it can give >>>>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139... >>>>> >>>>> >>>>> >>>>> a spin and let me know, I'd appreciate it. >>>>> >>>>> Probably easiest is just to check out and build >>>>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak >>>>> >>>>> On 10/20/11 11:49 AM, Brian Stansberry wrote: >>>>>> See [1] for a good discussion of this; last post points at a >>>>>> possible >>>>>> workaround. >>>>>> >>>>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966 >>>>>> On 10/19/11 6:26 PM, Stuart Douglas wrote: >>>>>>> It is probably to do with the number of tests in the test suite, as >>>>>>> now all the tests that were previously in preview and spec are >>>>>>> executed in one run. >>>>>>> >>>>>>> Stuart >>>>>>> >>>>>>> >>>>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote: >>>>>>> >>>>>>>> I agree that we should be closing the connection >>>>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/j... >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> is an example of where we handled that in AS6). Probably after we >>>>>>>> are >>>>>>>> done with the MBeanServerConnection. >>>>>>>> >>>>>>>> I created ARQ-632 for this. >>>>>>>> >>>>>>>> I was expecting the cause to be a regression but instead perhaps >>>>>>>> its >>>>>>>> the performance improvements (we are running unit tests quicker, >>>>>>>> causing more "jmx server connection timeout" threads to be >>>>>>>> created). >>>>>>>> Or, due to the number of tests in the testsuite now. >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Brian Stansberry"<brian.stansberry(a)redhat.com> >>>>>>>> To: jboss-as7-dev(a)lists.jboss.org >>>>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM >>>>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I >>>>>>>> synced with master yesterday >>>>>>>> >>>>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection() >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> is creating a javax.management.remote.JMXConnector but I don't see >>>>>>>> anything closing it. I'm not sure, but perhaps one of those is >>>>>>>> created >>>>>>>> per test method execution? >>>>>>>> >>>>>>>> I believe the "JMX server connection timeout 552" threads are >>>>>>>> created by >>>>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can >>>>>>>> see the >>>>>>>> relevant code in the stack trace. A quick look at that code leads >>>>>>>> me to >>>>>>>> believe those threads will by default live for 120 secs after the >>>>>>>> last >>>>>>>> request on the connection. >>>>>>>> >>>>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote: >>>>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote: >>>>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote: >>>>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun >>>>>>>>>>> 1.6.0_24-b07. >>>>>>>>>>> >>>>>>>>>>> Jesper also got the failures with jdk 6 u26. >>>>>>>>>>> >>>>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could >>>>>>>>>>> try >>>>>>>>>>> that >>>>>>>>>>> or we could start comparing environment settings. >>>>>>>>>>> >>>>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m >>>>>>>>>>> -XX:MaxPermSize=128m" >>>>>>>>>>> >>>>>>>>>>> I don't have that set locally. I'll try that next... >>>>>>>>>> >>>>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set. >>>>>>>>>> >>>>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that >>>>>>>>>> makes >>>>>>>>>> a diff. >>>>>>>>> >>>>>>>>> Same failure still (with MAVEN_OPTS set as above and using >>>>>>>>> jdk1.6.0_24). >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Scott >>>>>>>>>>> >>>>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote: >>>>>>>>>>>> I don't know about the test issues that Ondrej is running >>>>>>>>>>>> into, >>>>>>>>>>>> but the >>>>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : >>>>>>>>>>>> Cannot >>>>>>>>>>>> create native thread) were encountered once by Carlo (or on >>>>>>>>>>>> our >>>>>>>>>>>> Hudson >>>>>>>>>>>> instance, don't remember). But I haven't seen them after >>>>>>>>>>>> that. It >>>>>>>>>>>> might >>>>>>>>>>>> be environment specific. What JDK vendor/version? And are you >>>>>>>>>>>> running >>>>>>>>>>>> into this always? >>>>>>>>>>>> >>>>>>>>>>>> -Jaikiran >>>>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote: >>>>>>>>>>>>> How did these testsuite failures slip in? From my own >>>>>>>>>>>>> experience >>>>>>>>>>>>> running the updated testsuite, it looks like we now have an >>>>>>>>>>>>> allTests >>>>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we >>>>>>>>>>>>> merged some >>>>>>>>>>>>> changes and only ran the allTests profile. I think that we >>>>>>>>>>>>> need to >>>>>>>>>>>>> continue to use -DallTests instead (that includes the >>>>>>>>>>>>> integration >>>>>>>>>>>>> tests). Does anyone know more? >>>>>>>>>>>>> >>>>>>>>>>>>> I'm not sure why some people aren't seeing the failures >>>>>>>>>>>>> but it >>>>>>>>>>>>> seems we >>>>>>>>>>>>> need to track backwards from the symptoms (listed below) or >>>>>>>>>>>>> rollback >>>>>>>>>>>>> some changes until it goes away. >>>>>>>>>>>>> >>>>>>>>>>>>> If anyone is working on tracking the cause down, please speak >>>>>>>>>>>>> up. I can >>>>>>>>>>>>> jump in but don't want to duplicate effort. >>>>>>>>>>>>> >>>>>>>>>>>>> Scott >>>>>>>>>>>>> >>>>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote: >>>>>>>>>>>>>> The hudson AS7 test just finished with all integration tests >>>>>>>>>>>>>> passing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> When you say that your using a fresh master, is that a new >>>>>>>>>>>>>> AS7 >>>>>>>>>>>>>> source >>>>>>>>>>>>>> tree or an existing one freshly synced with AS7 master? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote: >>>>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>>>>>>>>>>>>>>> I have a fresh master and testsuite runs fine. >>>>>>>>>>>>>>>> Except 2 failures (for which I also seek comments): >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> server.log: >>>>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I didn't see that failure. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Which folder are you starting the unit test from and >>>>>>>>>>>>>>> what is >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> command line? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the >>>>>>>>>>>>>>> as7/testsuite/integration folder. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test >>>>>>>>>>>>>>> (was using >>>>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone >>>>>>>>>>>>>>> changed >>>>>>>>>>>>>>> that but >>>>>>>>>>>>>>> the test is still in queue. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition >>>>>>>>>>>>>>>> caused by >>>>>>>>>>>>>>>> improper use of deployer API, but giving it some time >>>>>>>>>>>>>>>> after >>>>>>>>>>>>>>>> deploy >>>>>>>>>>>>>>>> didn't help. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ondra >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run >>>>>>>>>>>>>>>>> fine for a >>>>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new >>>>>>>>>>>>>>>>> native thread" >>>>>>>>>>>>>>>>> errors start showing up in the server.log. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection >>>>>>>>>>>>>>>>> timeout NNN" >>>>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and >>>>>>>>>>>>>>>>> do a >>>>>>>>>>>>>>>>> "mvn clean >>>>>>>>>>>>>>>>> install -DallTests". >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Scott >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> > >

How did these testsuite failures slip in? From my own experience running the updated testsuite, it looks like we now have an allTests profile that doesn't run the integration tests. Perhaps we merged some changes and only ran the allTests profile. I think that we need to continue to use -DallTests instead (that includes the integration tests). Does anyone know more?

I'm not sure why some people aren't seeing the failures but it seems we need to track backwards from the symptoms (listed below) or rollback some changes until it goes away. If anyone is working on tracking the cause down, please speak up. I can jump in but don't want to duplicate effort. Scott On 10/18/2011 02:15 PM, Scott Marlow wrote: > The hudson AS7 test just finished with all integration tests passing. > > When you say that your using a fresh master, is that a new AS7 source > tree or an existing one freshly synced with AS7 master? > > > On 10/18/2011 12:26 PM, Scott Marlow wrote: >> On 10/18/2011 08:57 AM, Ondřej Žižka wrote: >>> I have a fresh master and testsuite runs fine. >>> Except 2 failures (for which I also seek comments): >>> >>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess >>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase >>> >>> server.log: >>> http://pastebin.test.redhat.com/64949 >> >> I didn't see that failure. >> >> Which folder are you starting the unit test from and what is your >> command line? >> >> I was trying to use "mvn clean install -DallTests" in the >> as7/testsuite/integration folder. >> >> When I checked earlier, Hudson wasn't running the full test (was using >> -PallTests instead of -DallTests). Looks like someone changed that but >> the test is still in queue. >> >> >>> >>> Anyone seeing that? I thought it could be a race condition caused by >>> improper use of deployer API, but giving it some time after deploy >>> didn't help. >>> >>> Ondra >>> >>> >>> >>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400: >>> >>>> When running the as7/testsuite/integration tests, we run fine for a >>>> while but then "OutOfMemoryError: unable to create new native thread" >>>> errors start showing up in the server.log. >>>> >>>> Failing tests are here http://pastebin.com/D3SeJ2rP >>>> >>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN" >>>> threads is here http://pastebin.com/qDC52WJG >>>> >>>> To recreate change into as7/testsuite/integration and do a "mvn clean >>>> install -DallTests". >>>> >>>> Is anyone not seeing this after syncing with master? >>>> >>>> Scott >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev