Thank You so much for the response.
The main problem is what you mentioned above. i.e when this issue happens we DO NOT have 4*275 concurrent request (clients). This is why i really can't figure out how we are reaching this maximum "in use connection count".
We don't have a <blocking-timeout-millis> entry in the ds file, which means it is using the default (i.e 30 seconds).
We did 2 of the above which you mentioned as a workaround. i.e added more jboss instances, limited the number of web sessions. These changes did in fact improved our situation but we still reach the "maximum in use connection count".
Also i wanted to share this link with you - - https://issues.jboss.org/browse/JBAS-4047. I got this as a reply for my posting.
Like it is mentioned in the above Jboss URL:
“The PooledInvokerProxy lock is held while getting an available connection from the pool and while the connection is tested. If the connection is to a local (machine) process, it may not take long for the server ACK to arrive. If the connection is to a remote (machine) process, it could take longer depending on network latency (especially if a wan is between tiers). “
My thought is that when the ACK server call (since we are using PooledInvoker) isslow, many threads will wait for a connection.
Again thanks for your time.
Anil Mathew