We have a very weird issue, we are fighting with for about a week already.
We moved from a single machine environment into a more professional setup, with hardware lb in front and have some issues.
Previous setup was : httpd (ssl offloading) -> jboss -> db
Now loadbalancer (ssl) -> jboss - db.
>From time to time we have a background process that consumes a lot of cpu and lasts for some minutes. Occasionally the whole app would stop. By stop I mean that the app is there, but reacting extremely slow. However, we never have had more than 30% cpu usage (4 code virtual machine). What we do see, is that we have almost 500 running http connector threads, however, most of them are in socket.read, socket.write, and not within our code.
I started to take regular thread dumps, and wrote a little utility that compares them, and I see that we have a lot of http connector threads that remain in same state over multiple hours:
"http--0.0.0.0-8443-18" daemon prio=10 tid=0x0000000002efb800 nid=0x10b2 runnable [0x00007fca4fcbd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:731)
at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:368)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:838)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:722)
We have 105 threads in this state over multiple hours (even during normal app operation).
Question: Could it be that we have connection leakage through keep-alive connections? Is there are possibility to reduce them?
Maybe any other points?
Thanks in advance