<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="https://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
HornetQ with CORE API result in HTTP Thread Waiting
</h3>
<span style="margin-bottom: 10px;">
created by <a href="https://community.jboss.org/people/walanj.amol">Amol Walanj</a> in <i>Performance Tuning</i> - <a href="https://community.jboss.org/message/775029#775029">View the full discussion</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p><strong>Title : HornetQ with CORE API result in HTTP Thread Waiting</strong></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Product Details :</strong></p><p>Jboss 5.1 Native</p><p>HornetQ 2.2.5 Final</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Issue :</strong></p><p>Jboss with HornetQ core api not able to scale and hence degrade the performance of the system as HTTP thread goes into waiting state</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Thread Dump :</strong></p><p>"http-0.0.0.0-8180-48" daemon prio=10 tid=0x00007fdd6800e000 nid=0x5ddf waiting on condition [0x00007fdd576f4000]</p><p>   java.lang.Thread.State: WAITING (parking)</p><p> at sun.misc.Unsafe.park(Native Method)</p><p> - parking to wait for  <0x00007fdfc10ab010> (a java.util.concurrent.Semaphore$NonfairSync)</p><p> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)</p><p> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)</p><p> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)</p><p> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)</p><p> at java.util.concurrent.Semaphore.acquire(Semaphore.java:441)</p><p> at org.hornetq.core.client.impl.ClientProducerCreditsImpl.acquireCredits(ClientProducerCreditsImpl.java:74)</p><p> at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:305)</p><p> at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:135)</p><p> at com.demo.services.MyServiceImpl.sendMessage( MyServiceImpl.java:174)</p><p> at com.demo.services.MyServiceImpl.placementRequest( MyServiceImpl.java:103)</p><p> at sun.reflect.GeneratedMethodAccessor285.invoke(Unknown Source)</p><p> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)</p><p> at java.lang.reflect.Method.invoke(Method.java:597)</p><p> at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)</p><p> at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)</p><p> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167)</p><p> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94)</p><p> at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)</p><p> at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94)</p><p> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)</p><p> - locked <0x00007fe54ee74e80> (a org.apache.cxf.phase.PhaseInterceptorChain)</p><p> at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:122)</p><p> at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:211)</p><p> at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)</p><p> at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:154)</p><p> at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:129)</p><p> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:187)</p><p> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:110)</p><p> at com.demo.services.MyCXFServlet.doPost( MyCXFServlet.java:33)</p><p> at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)</p><p> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:166)</p><p> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)</p><p> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)</p><p> at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)</p><p> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)</p><p> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)</p><p> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)</p><p> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)</p><p> at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)</p><p> at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)</p><p> at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)</p><p> at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)</p><p> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)</p><p> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)</p><p> at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)</p><p> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)</p><p> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)</p><p> at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:905)</p><p> at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:592)</p><p> at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2036)</p><p> at java.lang.Thread.run(Thread.java:662)</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Details :</strong></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Requirement :</strong></p><p>A REST(XmlOverHttp) based web service which respond to request, After responding to request, notification will be generated using the sent response and the notification will be sent  to billing system. </p><p>Delay of 10-15 min is ok for sending notifications but there has to be guarranty that every response sent should send notification. Because notifications are related to revenue / billing. </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Implementation and Observation :</strong></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"><strong> </strong> </p><p>In order to send notification after responding to webservice request, we used JMS. To make notification sending asynchronouts, before returning the response our serviceImpl class is putting response on jms queue, </p><p>later the consumer will prepare notification using the response received. </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>To achieve this initially we started with jboss Messaging using JMS API with persistence</p><p>Using this with the session caching we achieve TPS of around 1100</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Here we found JMS send is taking time in send so the performance of normal flow getting impacted.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Then we moved to hornetQ , and using  JMS API in same JBOSS we almost find the same TPS.</p><p>Latter we tried with standalone HorentQ. Here we reached TPS Up to 1300</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>After that we came to know hornetQ core Api are faster than JMS API.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>So we moved our code to HornetQ core API.</p><p>But we find that our TPS gets degraded to 200-250 using same.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Code Snippet :</strong></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Lifecycle Class which initialize the producer and consumer</strong></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>  </p><p>    public static void init() {</p><p>        clearCache()</p><p>        final MyConfiguration MyCfg = MyCfgMgr.getAppCfg();</p><p> mySession mySession = null;</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>        try {</p><p> TransportConfiguration config = new TransportConfiguration(</p><p> NettyConnectorFactory.class.getName());</p><p> locator = HornetQClient.createServerLocatorWithoutHA(config);</p><p> locator.setBlockOnAcknowledge(false);</p><p> locator.setBlockOnDurableSend(false);</p><p> myfactory = locator.createSessionFactory();</p><p> }catch(HornetQException je){</p><p> LOGGER.log("HornetQException ", je);</p><p>        }catch(Exception je){</p><p> LOGGER.log("Exception  ", je);</p><p>        }</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>        for (int i = 0; i < myCfg.getMDBPoolSize(); i++) {</p><p> try {</p><p> ClientSession session = myfactory.createSession(true , true,1);</p><p> session.start();</p><p> ClientProducer clientProducer = session.createProducer("myAddress");</p><p> mySession = new mySession(session, clientProducer ) ;</p><p> }catch(HornetQException je){</p><p> LOGGER.log("HornetQException ", je);</p><p> }catch(Exception je){</p><p> LOGGER.log("Exception ", je);</p><p> }</p><p> mySessionList.add(mySession);</p><p>        }</p><p>        initmyConsumer();</p><p>    }</p><p>    </p><p>    public static void initmyConsumer(){</p><p>        try{</p><p> TransportConfiguration config = new TransportConfiguration(NettyConnectorFactory.class.getName());</p><p> consumerLocator = HornetQClient.createServerLocatorWithoutHA(config);</p><p> consumerLocator.setBlockOnAcknowledge(false);</p><p> consumerLocator.setBlockOnDurableSend(false);</p><p> hFactory = consumerLocator.createSessionFactory(); </p><p>            consumerSession = hFactory.createSession(true,true,1);</p><p> consumerSession.start();</p><p> for(int i = 1 ; i < myCfg.getMDBPoolSize() ; i++){</p><p> ClientConsumer consumer = consumerSession.createConsumer("myQueue"); </p><p>                consumer.setMessageHandler(new MyMsgConsumer());</p><p> consumerList.add(consumer);</p><p> }</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>        }catch(HornetQException je){</p><p> LOGGER.log("HornetQException  ", je);</p><p>        }catch(Exception je){</p><p> LOGGER.log("Exception  ", je);</p><p>        }</p><p>    }</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>  public static void clearCache()</p><p>    {</p><p> LOGGER.log("clearCache PdnSessionCache : "+pdnSessionList.size());</p><p>        for(PdnSession pdnSession : pdnSessionList){</p><p> try {</p><p> pdnSession.getClientProducer().close(); </p><p>                pdnSession.getSession().close();</p><p> } catch (HornetQException e) {</p><p> LOGGER.log("HornetQException while closing Sessions from cache ", e);</p><p> }</p><p>        }</p><p>        if (locator != null) {</p><p> locator.close();</p><p>        }</p><p>        if (pdnfactory != null) {</p><p> pdnfactory.close();</p><p>        }</p><p> pdnSessionList.clear();</p><p>        </p><p>        try {</p><p> clearConsumer();</p><p>        } catch (HornetQException e) {</p><p> LOGGER.log("HornetQException while closing ClientSession ", e);</p><p>        }</p><p>        </p><p>    }</p><p>    </p><p>    public static void clearConsumer() throws HornetQException</p><p>    {</p><p> for(ClientConsumer c : consumerList){</p><p> c.close();</p><p>        }</p><p> consumerList.clear();</p><p>        if (consumerLocator != null) {</p><p> consumerLocator.close();</p><p>        }</p><p>        if (consumer != null) {</p><p> consumer.close();</p><p>        }</p><p>        if (consumerSession != null) {</p><p> consumerSession.close();</p><p>        }</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>        if (hFactory != null) {</p><p> hFactory.close();</p><p>        }</p><p>    }</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>MyMsgConsumer class is message consumer and is responsible for handling messages and sending notifications. </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Message Creation and sending :</strong></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Class Name : MyServiceImpl.java</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>private void sendMessage(final String res){</p><p>        try{</p><p>         // MySession is wrapper class holding ClientSession and ClientProducer as its members</p><p> MySession mySession = MyGlobals.getmyJmsSession(); </p><p>            ClientMessage msg = mySession.getSession().createMessage(org.hornetq.api.core.Message.TEXT_TYPE, true);            </p><p>            msg.getBodyBuffer().writeString(res);</p><p> mySession.getClientProducer().send(msg);</p><p> MyGlobals.returnmyJmsSession(mySession);</p><p> }catch(HornetQException je){</p><p> LOGGER.log("HornetQException ", je);</p><p>        }catch(Exception je){</p><p> LOGGER.log("Exception  ", je);</p><p>        }</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>    }</p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Reply to this message by <a href="https://community.jboss.org/message/775029#775029">going to Community</a></p>
<p style="margin: 0;">Start a new discussion in Performance Tuning at <a href="https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2078">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>