[jboss-jira] [JBoss JIRA] Commented: (JBAS-4586) unified invoker causes delays under load

[Galder Zamarreno (JIRA)]

    [ https://jira.jboss.org/jira/browse/JBAS-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12424084#action_12424084 ] 

Galder Zamarreno commented on JBAS-4586:
----------------------------------------

Troy, it's generally a good practice to load test your application 
before putting it in production. This avoids you suprises like the 
one you had and helps you tweak the defaults based on the 
machine(s) where it's running, the application...etc.

Finally, whenever you get hang ups, or server slow downs, 
generate thread dumps and inspect the GC logs.

There're tools out there that can help you diagnose these.

> unified invoker causes delays under load
> ----------------------------------------
>
>                 Key: JBAS-4586
>                 URL: https://jira.jboss.org/jira/browse/JBAS-4586
>             Project: JBoss Application Server
>          Issue Type: Sub-task
>      Security Level: Public(Everyone can see) 
>          Components: Remoting
>    Affects Versions: JBossAS-4.2.0.GA, JBossAS-4.2.1.GA
>         Environment: Linux kernel 2.6.21,  Java 1.5.0_12, JBoss 4.2.0.GA
>            Reporter: Troy Bowman
>            Assignee: Ron Sigal
>
> We tested Jboss 4.2.0.GA and found it worthy for production.  When we actually did put it in production, and more than about 300 people were using the application simultaneously, people complained that the application was extremely slow.  Some commands, which issued several rmi invocations, would take several minutes.  There was not a heavy load on the servers, however.  It seemed like every request was just waiting for something to timeout, or that something was causing deadlock in a synchronized  block.
> I analyzed the problem by both watching the jboss server.log, and also using tcpdump to watch network traffic.  I noticed big pauses where neither the server, client, or network was doing anything at all.  I made tcpdump have a zero snaplen and read through each packet trying to find what exactly it was doing.  I followed the process from ports 1099(naming) to 1098(rmi) to 4446(unified invoker).  Right when the client would invoke the stub it had from rmi, it'd sit there for 12 to 14 seconds.  I looked at this from other commands and indeed, it was waiting for the invoker.
> I started googling for hangs with the jboss invoker and stumbled across some bugs, and found these:
> http://jira.jboss.org/jira/browse/JBREM-203?decorator=printable
> http://jira.jboss.org/jira/browse/JBREM-167?page=all&decorator=printable
> http://jira.jboss.com/jira/browse/JBREM-165;jsessionid=4A6E5196E7A1A78EAFC58E899A40603D?page=all
> They're all old bugs, but at least it is obvious that the unified invoker had problems in the past with hanging while marshalling/unmarshalling with jboss's custom (un)marshallers.  My suspicion was that they may have solved the problem with the hack back then, but it is probably showing its ugly head when the server is under a heavy load.
> In searching for more stuff, I found that the readme for jboss-4.2.0.GA said that "The default invoker for EJBs has been changed from the rmi-invoker to the unified-invoker, provided by JBoss Remoting".  This got me very susupicious of it, since it was definitely the invoker that was sitting there for around 15 seconds on every request when the server had a heavier load.  I looked at the change to the standardjboss.xml:
> http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/branches/Branch_4_2/server/src/etc/conf/default/standardjboss.xml/?revision=59446&r1=59446&r2=59445
> I changed the proxy bindings from the unified invoker back to the rmi invoker in standardjboss.xml, and the problem went away.  Invocations now go through port 4444, and are instantaneous.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira