[jboss-user] Strange thread behavior in tomcat/jboss 7

[Leon Rosenberg]

Hello,

We have a very weird issue, we are fighting with for about a week already.
We moved from a single machine environment into a more professional setup,
with hardware lb in front and have some issues.
Previous setup was : httpd (ssl offloading) -> jboss -> db
Now loadbalancer (ssl) -> jboss - db.

>From time to time we have a background process that consumes a lot of cpu
and lasts for some minutes. Occasionally the whole app would stop. By stop
I mean that the app is there, but reacting extremely slow. However, we
never have had more than 30% cpu usage (4 code virtual machine). What we do
see, is that we have almost 500 running http connector threads, however,
most of them are in socket.read, socket.write, and not within our code.

I started to take regular thread dumps, and wrote a little utility that
compares them, and I see that we have a lot of http connector threads that
remain in same state over multiple hours:

"http--0.0.0.0-8443-18" daemon prio=10 tid=0x0000000002efb800 nid=0x10b2
runnable [0x00007fca4fcbd000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:731)
at
org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:368)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:838)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:722)

We have 105 threads in this state over multiple hours (even during normal
app operation).

Question: Could it be that we have connection leakage through keep-alive
connections? Is there are possibility to reduce them?
I understand that jboss have the same apache 11 connector, however all
extended configuration options are removed from the doc:
http://docs.jboss.org/jbossweb/7.0.x/config/http.html

Maybe any other points?

Thanks in advance
Leon

JBoss AS7, Java 1.7, ubuntu 10,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20130206/fbd639f8/attachment.html