[jboss-jira] [JBoss JIRA] Created: (JBMESSAGING-1210) Simple messaging test unable to drive CPU utilization better than 20%

[P Duffy (JIRA)]

Simple messaging test unable to drive CPU utilization better than 20%
---------------------------------------------------------------------

                 Key: JBMESSAGING-1210
                 URL: http://jira.jboss.com/jira/browse/JBMESSAGING-1210
             Project: JBoss Messaging
          Issue Type: Bug
    Affects Versions: 1.3.0.GA
         Environment: EAP 4.2 running JBoss Messaging 1.3.  Solaris 10 running 
on Sun T2000 server.  Note that T2000 is able to run 32 threads 
concurrently.  We routinely run various tests to confirm AS is fully 
utilizing available CPU.


            Reporter: P Duffy
         Assigned To: Tim Fox


After several months, we have been unable to resolve a JBM performance issue. Recent JBoss consultant visit has not helped.

We narrowed down our performance test to the following:
- several instances of a multi threaded test client sending a simple 
ObjectMessage into a single AS instance.
- a single MDB is deployed in the AS instance.  This MDB consumes the 
message and throws it away (does nothing with it).

When this test is run, the AS instance indicates messages are backing up 
in the Q, yet the clients are unable to drive the CPU past 20% utilization.

We've tried the following tuning adjustments:

(1) maxSession Annotation on MDB:
    @ActivationConfigProperty(propertyName = "maxSession", propertyValue
= "300")
 
(2) maxSize in server/default/deploy/ejb3-interceptors-aop.xml, on
Message Driven Bean:
         @org.jboss.annotation.ejb.PoolClass
(value=org.jboss.ejb3.StrictMaxPool.class, maxSize=300, timeout=10000)
	
(3) maxPoolSize and numAcceptThreads in
server/default/deploy/jboss-messaging.sar/remoting-service.xml
   <attribute name="maxPoolSize" isParam="true">300</attribute>
   <attribute name="numAcceptThreads" isParam="true">50</attribute>

(4) maxPoolSize in ThreadPool in /server/default/conf/jboss-service.xml
	<attribute name="MaximumPoolSize">800</attribute>


Specific documentation re: JBM performance tuning is utterly lacking. 
We have tried to piece together various shreds of configuration from 
support cases, consultant comments, etc.  to no avail.

We need SPECIFIC direction from JBoss re: how to properly configure 
messaging to fully utilize available CPU. Else identify underlying bugs 
and resolve same.

P.S.  We have performed some examination of the JBM 1.3 code.   With the 
huge caveat that we are not expert re: JBM internals, we have identified
one location at which 100+ threads appear to be waiting on a single 
threaded synchronization point.  See below.


-----Original Message-----
From: Mickael Graham (mickaelg)
Sent: Monday, December 17, 2007 5:21 PM
To: Nick Culpepper (nculpepp)
Subject: Re: Thread dump for MDBEcho test

Yes.  Heaps of threads are blocked on this:

at
org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServe
rInvocationHandler.java:105)
	- waiting to lock <0xfffffffeb17129b8> (a java.lang.Class)

This is where the performance problem is.  Looking at the JBoss
messaging code:

  public Object invoke(InvocationRequest invocation) throws Throwable
    {
       if (trace) { log.trace("invoking " + invocation); }

       synchronized (JMSServerInvocationHandler.class)
       {
          if (closed)
          {
             throw new MessagingJMSException("Cannot handle invocation
since messaging server is not active (it is either starting up or
shutting down)");
          }

          RequestSupport request =
(RequestSupport)invocation.getParameter();

          if (request instanceof
ConnectionFactoryCreateConnectionDelegateRequest)
          {
             //Create connection request

             ConnectionFactoryCreateConnectionDelegateRequest cReq =

(ConnectionFactoryCreateConnectionDelegateRequest)request;

             String remotingSessionId = cReq.getRemotingSessionID();

             ServerInvokerCallbackHandler callbackHandler = null;
             synchronized(callbackHandlers)
             {
                callbackHandler =
(ServerInvokerCallbackHandler)callbackHandlers.get(remotingSessionId);
             }
             if (callbackHandler != null)
             {
                log.debug("found calllback handler for remoting session
" + Util.guidToString(remotingSessionId));

                cReq.setCallbackHandler(callbackHandler);
             }
             else
             {
                throw new IllegalStateException("Cannot find callback
handler " +
                                                "for session id " +
remotingSessionId);
             }
          }

          return request.serverInvoke();
       }
    }

Basically this is for delivering I believe and that looks single
threaded.  Perhaps once its delivered, it will not be single threaded.



-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira