possibilities:
- connection pool leak,
http://wiki.jboss.org/wiki/Wiki.jsp?page=CanJBossTellMeWhenIDontCloseACon..., fix your
app
- another resource pool in the system is constraining the number of concurrent
connections/threads.
- JBoss not providing the required number of connections in the pool, check bug database,
if none match, raise a defect will jboss,
- db server not configured to allow more than 100 connections, speak to dba as suggested,
you've already taken thread dumps and found not all threads are waiting for new db
cons. points towards a leak or some other thread pool constraint
get some stats from
http://wiki.jboss.org/wiki/Wiki.jsp?page=CanJBossTellMeWhenIDontCloseACon...
the pause/hang is probably due to the server running out of threads to service the
requests (server.xml, maxThreads, default 200 - that needs to be at least 500 if you
expect 500 concurrent requests), once this happens there is an overflow number
(aceptCount, default 10) of connections that jboss will allow your stress tool to connect
to, which will just sit and wait to be serviced also.
as the connection wait timeout is 30 seconds, once the connection pool is exhausted
requests will wait 30 seconds until they fail. When you have >100 concurrent clients it
may take a long time for all those requests to timeout before the server will respond
again (the 200 seconds sounds reasonable). I'd suggest reducing the timeout to
something like 200 milliseconds if your db server is on a local gigabit LAN, any longer
and it's likely there is a problem in which case you'd want it to fail sooner
rather than later in tests. On a LAN you should be able to connect to your db server in a
couple of milliseconds.
monitor the number of concurrent connection to Jboss and db server on a 30 second interval
and note if/when the connections shoot up. take dumps immediately after, don't wait
for the hang.
does the same issue occur for 400,300,200,50 users after 1,2,3 hours? If it's a leak
it doesn't matter how many users necessarily, as it may just be a matter of time
before the connections run out.
I'd hope some of the above would get you closer to a diagnosis.
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4052756#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...