[jboss-jira] [JBoss JIRA] Commented: (JBAS-6665) Server-side thread count steadily increases during AS testsuite runs

[Richard Achmatowicz (JIRA)]

    [ https://jira.jboss.org/jira/browse/JBAS-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12459716#action_12459716 ] 

Richard Achmatowicz commented on JBAS-6665:
-------------------------------------------

Based on comments from Ron and Jesper, I did another run of the jboss-all-config-tests with reduced thread pool parameters for JCA and Remoting:

jca-jboss-beans.xml:
poolsize=25           // max thread pool size of 25
keepAliveTime=30 *1000)   // reclaim threads if not used for 30 secs

remoting-jboss-beans.xml:
maxPoolSize=30    // max thread pool size of 30
idleTimeout=60     // reclaim threads if not used for 60 secs

These two changes drastically reduced the number of threads active in the server at any given time, as expected.

Before the change: (see memoryUsage.annotated.log)
start of testsuite:       active threads: 128, daemon threads: 76, peak thread count: 128,  tolal threads allocated: 146
end of testsuite:        active threads: 325, daemon threads: 237, peak thread count: 347,  tolal threads allocated: 1961

After the change: (see memoryUsage.annotated.2.log)
start of testsuite:       active threads: 125, daemon threads: 75, peak thread count: 125,  tolal threads allocated: 141
end of testsuite:        active threads: 234, daemon threads: 161, peak thread count: 257,  tolal threads allocated: 1462

Thus, it appears that large pool sizes (300 as opposed to 30) and no reaping of idle threads (as opposed to reaping every 60 seconds) accounts for the creeping increase in active thread counts.

As mentioned, on certain platforms (HPUX), these large thread counts were causing the testsuite to run out of threads and fail. 

Should the values for these settings in the 'all' configuration be revised to allow for fewer testsuite failures? For example, to at least turn on reaping of idle threads in Remoting? 








> Server-side thread count steadily increases during AS testsuite runs 
> ---------------------------------------------------------------------
>
>                 Key: JBAS-6665
>                 URL: https://jira.jboss.org/jira/browse/JBAS-6665
>             Project: JBoss Application Server
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Test Suite
>    Affects Versions: JBossAS-5.0.1.GA
>         Environment: AS testsuite - not particular to any platform
>            Reporter: Richard Achmatowicz
>            Assignee: Richard Achmatowicz
>            Priority: Minor
>         Attachments: memoryUsage.annotated.log
>
>
> (from an original email)
> > Hi Rajesh
> >
> > Just wanted to let you know about something concerning the AS testsuite.
> >
> > I've been having real problems getting the AS 5.0.0 (and 5.0.1) testsuite to run on HPUX without running out of native threads. It won't run to completion without getting an "OutOfMemoryError: unable to create native thread" exception. So I decided to try to find out how many threads are active in the JVM at any one time during a testsuite run.
> >
> > I wrote a small modification to XMLJUnitMultipleResultFormatter which gets called before and after every testsuite; it prints out the following statistics:
> > <test name>: ThreadCounts(current threads, daemon threads,peak thread count,total started), HeapSizes(init,used,committed,max)
> >
> > I'm seeing a strange creeping increase in the number of current threads active in the JVM over the life of the 'all' server execution, which is where i'm having the problem with native thread allocation. I would guess from this that some test cases are not releasing threads when finished, although this is only a guess.
> This is interesting. I would like to know if the HP UX maximum thread count is really low or the AS doesn't release some threads on undeploying the tests.
> Does a profiler show the list of all active threads when it runs out of active threads?
> A comparison with a run on RHEL with these metrics enabled will also tell us if we have a problem of threads being free'd on HPUX
> Can you open a JBAS and assign it to yourself with the below details and the attached log?
> There should be some interesting comments on this from dev. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira