[jboss-jira] [JBoss JIRA] Commented: (JBAS-4586) unified invoker causes delays under load

[Troy Bowman (JIRA)]

    [ https://jira.jboss.org/jira/browse/JBAS-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12422773#action_12422773 ] 

Troy Bowman commented on JBAS-4586:
-----------------------------------

Ron, thanks for your reply.   We're still using the RMI invoker since this bug was created a year ago.   My reflex is to be afraid to go back to the unified invoker, since there may be some other unreasonable default setting that may need to be tweaked that will affect production.  I am still interested in pursuing this issue.  I hope to be able to have a chance to test the maxPoolSize setting in a test environment soon.  But yes, apparently it is a configuration problem.  Indeed, it is not only that, but also lack of documentation about this, and possibly a lack of more reasonable defaults.  If I google for information on this problem, this bug is the first hit.  Users should not have to reverse-engineer a program to be able to figure out why it is not performing.

> unified invoker causes delays under load
> ----------------------------------------
>
>                 Key: JBAS-4586
>                 URL: https://jira.jboss.org/jira/browse/JBAS-4586
>             Project: JBoss Application Server
>          Issue Type: Sub-task
>      Security Level: Public(Everyone can see) 
>          Components: Remoting
>    Affects Versions: JBossAS-4.2.0.GA, JBossAS-4.2.1.GA
>         Environment: Linux kernel 2.6.21,  Java 1.5.0_12, JBoss 4.2.0.GA
>            Reporter: Troy Bowman
>            Assignee: Ron Sigal
>
> We tested Jboss 4.2.0.GA and found it worthy for production.  When we actually did put it in production, and more than about 300 people were using the application simultaneously, people complained that the application was extremely slow.  Some commands, which issued several rmi invocations, would take several minutes.  There was not a heavy load on the servers, however.  It seemed like every request was just waiting for something to timeout, or that something was causing deadlock in a synchronized  block.
> I analyzed the problem by both watching the jboss server.log, and also using tcpdump to watch network traffic.  I noticed big pauses where neither the server, client, or network was doing anything at all.  I made tcpdump have a zero snaplen and read through each packet trying to find what exactly it was doing.  I followed the process from ports 1099(naming) to 1098(rmi) to 4446(unified invoker).  Right when the client would invoke the stub it had from rmi, it'd sit there for 12 to 14 seconds.  I looked at this from other commands and indeed, it was waiting for the invoker.
> I started googling for hangs with the jboss invoker and stumbled across some bugs, and found these:
> http://jira.jboss.org/jira/browse/JBREM-203?decorator=printable
> http://jira.jboss.org/jira/browse/JBREM-167?page=all&decorator=printable
> http://jira.jboss.com/jira/browse/JBREM-165;jsessionid=4A6E5196E7A1A78EAFC58E899A40603D?page=all
> They're all old bugs, but at least it is obvious that the unified invoker had problems in the past with hanging while marshalling/unmarshalling with jboss's custom (un)marshallers.  My suspicion was that they may have solved the problem with the hack back then, but it is probably showing its ugly head when the server is under a heavy load.
> In searching for more stuff, I found that the readme for jboss-4.2.0.GA said that "The default invoker for EJBs has been changed from the rmi-invoker to the unified-invoker, provided by JBoss Remoting".  This got me very susupicious of it, since it was definitely the invoker that was sitting there for around 15 seconds on every request when the server had a heavier load.  I looked at the change to the standardjboss.xml:
> http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/branches/Branch_4_2/server/src/etc/conf/default/standardjboss.xml/?revision=59446&r1=59446&r2=59445
> I changed the proxy bindings from the unified invoker back to the rmi invoker in standardjboss.xml, and the problem went away.  Invocations now go through port 4444, and are instantaneous.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira