[jboss-jira] [JBoss JIRA] Closed: (JBMESSAGING-1210) Simple messaging test unable to drive CPU utilization better than 20%

Simple messaging test unable to drive CPU utilization better than 20% --------------------------------------------------------------------- Key: JBMESSAGING-1210 URL: http://jira.jboss.com/jira/browse/JBMESSAGING-1210 Project: JBoss Messaging Issue Type: Bug Affects Versions: 1.3.0.GA Environment: EAP 4.2 running JBoss Messaging 1.3. Solaris 10 running on Sun T2000 server. Note that T2000 is able to run 32 threads concurrently. We routinely run various tests to confirm AS is fully utilizing available CPU. Reporter: P Duffy Assigned To: Tim Fox After several months, we have been unable to resolve a JBM performance issue. Recent JBoss consultant visit has not helped. We narrowed down our performance test to the following: - several instances of a multi threaded test client sending a simple ObjectMessage into a single AS instance. - a single MDB is deployed in the AS instance. This MDB consumes the message and throws it away (does nothing with it). When this test is run, the AS instance indicates messages are backing up in the Q, yet the clients are unable to drive the CPU past 20% utilization. We've tried the following tuning adjustments: (1) maxSession Annotation on MDB: @ActivationConfigProperty(propertyName = "maxSession", propertyValue = "300") (2) maxSize in server/default/deploy/ejb3-interceptors-aop.xml, on Message Driven Bean: @org.jboss.annotation.ejb.PoolClass (value=org.jboss.ejb3.StrictMaxPool.class, maxSize=300, timeout=10000) (3) maxPoolSize and numAcceptThreads in server/default/deploy/jboss-messaging.sar/remoting-service.xml <attribute name="maxPoolSize" isParam="true">300</attribute> <attribute name="numAcceptThreads" isParam="true">50</attribute> (4) maxPoolSize in ThreadPool in /server/default/conf/jboss-service.xml <attribute name="MaximumPoolSize">800</attribute> Specific documentation re: JBM performance tuning is utterly lacking. We have tried to piece together various shreds of configuration from support cases, consultant comments, etc. to no avail. We need SPECIFIC direction from JBoss re: how to properly configure messaging to fully utilize available CPU. Else identify underlying bugs and resolve same. P.S. We have performed some examination of the JBM 1.3 code. With the huge caveat that we are not expert re: JBM internals, we have identified one location at which 100+ threads appear to be waiting on a single threaded synchronization point. See below. -----Original Message----- From: Mickael Graham (mickaelg) Sent: Monday, December 17, 2007 5:21 PM To: Nick Culpepper (nculpepp) Subject: Re: Thread dump for MDBEcho test Yes. Heaps of threads are blocked on this: at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServe rInvocationHandler.java:105) - waiting to lock <0xfffffffeb17129b8> (a java.lang.Class) This is where the performance problem is. Looking at the JBoss messaging code: public Object invoke(InvocationRequest invocation) throws Throwable { if (trace) { log.trace("invoking " + invocation); } synchronized (JMSServerInvocationHandler.class) { if (closed) { throw new MessagingJMSException("Cannot handle invocation since messaging server is not active (it is either starting up or shutting down)"); } RequestSupport request = (RequestSupport)invocation.getParameter(); if (request instanceof ConnectionFactoryCreateConnectionDelegateRequest) { //Create connection request ConnectionFactoryCreateConnectionDelegateRequest cReq = (ConnectionFactoryCreateConnectionDelegateRequest)request; String remotingSessionId = cReq.getRemotingSessionID(); ServerInvokerCallbackHandler callbackHandler = null; synchronized(callbackHandlers) { callbackHandler = (ServerInvokerCallbackHandler)callbackHandlers.get(remotingSessionId); } if (callbackHandler != null) { log.debug("found calllback handler for remoting session " + Util.guidToString(remotingSessionId)); cReq.setCallbackHandler(callbackHandler); } else { throw new IllegalStateException("Cannot find callback handler " + "for session id " + remotingSessionId); } } return request.serverInvoke(); } } Basically this is for delivering I believe and that looks single threaded. Perhaps once its delivered, it will not be single threaded.