[jboss-jira] [JBoss JIRA] (JBAS-9526) High CPU with lots of Waiting threads on JIoEndpoint$Worker on version 5.0.1

[Pavel Kouzmitcheu (JIRA)]

    [ https://issues.jboss.org/browse/JBAS-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133925#comment-13133925 ] 

Pavel Kouzmitcheu edited comment on JBAS-9526 at 11/25/15 12:43 PM:
--------------------------------------------------------------------

I have very similar issue. +JBoss 5.1.0.+

During the load test with >200 users per instance each instance run into high CPU load >99%.
The thread dump analysis and statistics for CPU usage per Application Threads revealed that most of CPU (>75%) is taken by org.apache.tomcat.util.net.JIoEndpoint$Worker threads. During the load test multiple new+ JIoEndpoint$Worker threads+ were created, and +each took from 1 to 9% of CPU+. Surprisingly, most of JIoEndpoint$Worker threads which were on high CPU usage were in Waiting state, which doesn't makes much sense to me.
To collect CPU per Application thread statistics I used the following linux command:

{code:java}
ps -mo pid,lwp,stime,time,%cpu,%mem -C java >>/shared/threads_per_cpu.$(date +"%F_%H-%M-%S")
{code}


Why JIoEndpoint$Worker Threads took so much of CPU? (Especially the Threads which were in Waiting state)
Is there any way to workaround this? (e.g. configuration change or Hotfix)

Dump example for the heavy CPU threads:

~"http-0.0.0.0-8280-25" daemon prio=10 tid=0x00002b4f7d5ee800 nid=0x6c05 in Object.wait() [0x00002b4f9d64b000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000078f5c3298> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:420)
        - locked <0x000000078f5c3298> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
        at java.lang.Thread.run(Thread.java:662)
~


was (Author: pkouzm):
I have very similar issue. +JBoss 5.1.0.+

During the load test with >200 users per instance each instance run into high CPU load >99%.
The thread dump analysis and statistics for CPU usage per Application Threads revealed that most of CPU (>75%) is taken by org.apache.tomcat.util.net.JIoEndpoint$Worker threads. During the load test multiple new+ JIoEndpoint$Worker threads+ were created, and +each took from 1 to 9% of CPU+. Surprisingly, most of JIoEndpoint$Worker threads which were on high CPU usage were in Waiting state, which doesn't makes much sense to me.
To collect CPU per Application thread statistics I used the following linux command:
~ps -mo pid,lwp,stime,time,%cpu,%mem -C java >>/shared/threads_per_cpu.$(date +"%F_%H-%M-%S")~

Why JIoEndpoint$Worker Threads took so much of CPU? (Especially the Threads which were in Waiting state)
Is there any way to workaround this? (e.g. configuration change or Hotfix)

Dump example for the heavy CPU threads:

"http-0.0.0.0-8280-25" daemon prio=10 tid=0x00002b4f7d5ee800 nid=0x6c05 in Object.wait() [0x00002b4f9d64b000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000078f5c3298> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:420)
        - locked <0x000000078f5c3298> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
        at java.lang.Thread.run(Thread.java:662)


> High CPU with lots of Waiting threads on JIoEndpoint$Worker on version 5.0.1
> ----------------------------------------------------------------------------
>
>                 Key: JBAS-9526
>                 URL: https://issues.jboss.org/browse/JBAS-9526
>             Project: Application Server 3  4  5 and 6
>          Issue Type: Feature Request
>            Reporter: shikhar khanna
>
> High CPU with lots of Waiting threads on JIoEndpoint$Worker on version 5.0.1
> - Running a 100 user load test on JBoss 5.0.1.
> - Getting High CPU 
> - Analyzed thread dumps and found almost 50-60% threads are waiting as per below snippet\
>      Thread Name : http-0.0.0.0-8080-8 State : in Object.wait() Java Stack at java.lang.Object.wait(Native Method) - waiting on [0x24d875d0] (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416) - locked [0x24d875d0] (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442) at java.lang.Thread.run(Thread.java:662)



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)