JBR HTTP Gateway Performance Issue ---------------------------------- Key: JBESB-2605 URL: https://jira.jboss.org/jira/browse/JBESB-2605 Project: JBoss ESB Issue Type: Bug Security Level: Public (Everyone can see) Components: Transports Environment: SOA-P 4.3 CP01 with this patch: https://enterprise.redhat.com/issue-tracker/?module=issues&action=vie... Reporter: Aaron Pestel Attachments: Deployments.zip, JbdsProjects.zip, SoapUIProjects.zip In testing a configuration for using the ESB to proxy web services, the performance overhead of the ESB seemed excessive (almost 10X) and it appears there may be memory leaks with the JBR router. The test case is as follows: Deploy a simple web service in a stand alone WS WAR. Use SoapUI to run a load test against that web service for 5 minutes with only one thread - just to get average response time of web service, not to test multithreaded throughput. Now, create an ESB service that has a JBR/HTTP gateway, InVM listener, and HttpRouter action to proxy the web service. Run the same load test (one thread for five minutes) against the JBR/HTTP ESB service which routes to WS service and returns response. Here are the findings of the test: Using SoapUI to hit the web service directly for five minutes results in an average response time from the web service of about 3.73 ms - not bad for an average laptop. Memory utilization seems to be constant through the entire test. Using SoapUI to hit the ESB service (that routes to the same web service as above) works fine for the first 1.5-2 minutes with an average response time of about 65ms. However, Java heap of SOA-P instance seems to grow constantly. At about 3-4 minutes into the test, response times of the ESB service are about 1-2 seconds each - probably due to GC thrashing. Slightly before the end of the test, SOA-P instance starts getting OutOfMemoryErrors (stack traces at bottom of this description). I'm attaching three files: JBDS_Projects.zip - contains two JBDS 2.0 projects (web service WAR project and ESB project) Deployments.zip - contains the actual .war and .esb file generated from the projects SoapUIProjects.zip - contains the SoapUI projects with load tests configured to hit the web service directly and through the ESB proxy I think two problems exist: 1.) Even for the first half of the load test, the ESB proxy has a response time of around 10x (depending on runs) of the direct WS invocation, seems it should be closer to 2-3x 2.) Seems to be some type of memory leak in JBR gateway that eventually leads to OutOfMemoryErrors Here are some of the stack traces that may or may not provide value in diagnosing this, I was a bit surprised to see Smooks in the stack trace when the only ESB components I'm using are JBR gateway, InVM listener, and HttpRouter action. 14:00:56,810 ERROR [JBossRemotingGatewayListener] JBoss Remoting Gateway failed to synchronously deliver message to target service [WSProxy:WSProxy]. java.lang.OutOfMemoryError: Java heap space at com.sun.org.apache.xpath.internal.VariableStack.<init>(VariableStack.java:91) at com.sun.org.apache.xpath.internal.XPathContext.<init>(XPathContext.java:419) at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.eval(XPathImpl.java:201) at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:275) at org.milyn.xml.XmlUtil.getNodeList(XmlUtil.java:366) at org.milyn.xml.XmlUtil.getNode(XmlUtil.java:384) at org.milyn.cdr.XMLConfigDigester.digestResourceConfig(XMLConfigDigester.java:205) at org.milyn.cdr.XMLConfigDigester.digestV10XSDValidatedConfig(XMLConfigDigester.java:171) at org.milyn.cdr.XMLConfigDigester.digestConfig(XMLConfigDigester.java:93) at org.milyn.cdr.XMLConfigDigester.digestConfig(XMLConfigDigester.java:68) at org.milyn.cdr.SmooksResourceConfigurationStore.registerResources(SmooksResourceConfigurationStore.java:230) at org.milyn.cdr.SmooksResourceConfigurationStore.registerInstalledResources(SmooksResourceConfigurationStore.java:167) at org.milyn.cdr.SmooksResourceConfigurationStore.<init>(SmooksResourceConfigurationStore.java:104) at org.milyn.container.standalone.StandaloneApplicationContext.<init>(StandaloneApplicationContext.java:51) at org.milyn.Smooks.<init>(Smooks.java:93) at org.milyn.Smooks.<init>(Smooks.java:129) at org.jboss.soa.esb.services.security.auth.ws.WSSecurityInfoExtractor.createSmooks(WSSecurityInfoExtractor.java:124) at org.jboss.soa.esb.services.security.auth.ws.WSSecurityInfoExtractor.<init>(WSSecurityInfoExtractor.java:67) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener$JBossRemotingMessageComposer.populateMessage(JBossRemotingGatewayListener.java:523) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener$JBossRemotingMessageComposer.populateMessage(JBossRemotingGatewayListener.java:487) at org.jboss.soa.esb.listeners.message.AbstractMessageComposer.compose(AbstractMessageComposer.java:76) at org.jboss.soa.esb.listeners.message.UncomposedMessageDeliveryAdapter.deliverSyncWithoutDecomposing(UncomposedMessageDeliveryAdapter.java:104) at org.jboss.soa.esb.listeners.message.UncomposedMessageDeliveryAdapter.deliverSync(UncomposedMessageDeliveryAdapter.java:86) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener.invoke(JBossRemotingGatewayListener.java:345) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:866) at org.jboss.remoting.transport.coyote.CoyoteInvoker.service(CoyoteInvoker.java:310) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:595) 14:00:58,075 ERROR [JBossRemotingGatewayListener] JBoss Remoting Gateway failed to synchronously deliver message to target service [WSProxy:WSProxy]. java.lang.OutOfMemoryError: Java heap space at java.io.BufferedWriter.<init>(BufferedWriter.java:87) at java.io.BufferedWriter.<init>(BufferedWriter.java:70) at java.io.PrintWriter.<init>(PrintWriter.java:109) at com.sun.org.apache.xml.internal.utils.DefaultErrorHandler.<init>(DefaultErrorHandler.java:65) at com.sun.org.apache.xpath.internal.XPath.<init>(XPath.java:154) at com.sun.org.apache.xpath.internal.XPath.<init>(XPath.java:198) at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.eval(XPathImpl.java:193) at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:275) at org.milyn.xml.XmlUtil.getNodeList(XmlUtil.java:366) at org.milyn.xml.XmlUtil.getNode(XmlUtil.java:384) at org.milyn.cdr.XMLConfigDigester.digestResourceConfig(XMLConfigDigester.java:206) at org.milyn.cdr.XMLConfigDigester.digestV10XSDValidatedConfig(XMLConfigDigester.java:171) at org.milyn.cdr.XMLConfigDigester.digestConfig(XMLConfigDigester.java:93) at org.milyn.cdr.XMLConfigDigester.digestConfig(XMLConfigDigester.java:68) at org.milyn.cdr.SmooksResourceConfigurationStore.registerResources(SmooksResourceConfigurationStore.java:230) at org.milyn.cdr.SmooksResourceConfigurationStore.registerInstalledResources(SmooksResourceConfigurationStore.java:167) at org.milyn.cdr.SmooksResourceConfigurationStore.<init>(SmooksResourceConfigurationStore.java:104) at org.milyn.container.standalone.StandaloneApplicationContext.<init>(StandaloneApplicationContext.java:51) at org.milyn.Smooks.<init>(Smooks.java:93) at org.milyn.Smooks.<init>(Smooks.java:129) at org.jboss.soa.esb.services.security.auth.ws.WSSecurityInfoExtractor.createSmooks(WSSecurityInfoExtractor.java:124) at org.jboss.soa.esb.services.security.auth.ws.WSSecurityInfoExtractor.<init>(WSSecurityInfoExtractor.java:67) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener$JBossRemotingMessageComposer.populateMessage(JBossRemotingGatewayListener.java:523) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener$JBossRemotingMessageComposer.populateMessage(JBossRemotingGatewayListener.java:487) at org.jboss.soa.esb.listeners.message.AbstractMessageComposer.compose(AbstractMessageComposer.java:76) at org.jboss.soa.esb.listeners.message.UncomposedMessageDeliveryAdapter.deliverSyncWithoutDecomposing(UncomposedMessageDeliveryAdapter.java:104) at org.jboss.soa.esb.listeners.message.UncomposedMessageDeliveryAdapter.deliverSync(UncomposedMessageDeliveryAdapter.java:86) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener.invoke(JBossRemotingGatewayListener.java:345) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:866) at org.jboss.remoting.transport.coyote.CoyoteInvoker.service(CoyoteInvoker.java:310) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) 14:01:01,037 ERROR [JBossRemotingGatewayListener] JBoss Remoting Gateway failed to synchronously deliver message to target service [WSProxy:WSProxy]. java.lang.OutOfMemoryError: Java heap space at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:44) at java.lang.StringBuilder.<init>(StringBuilder.java:69) at java.lang.Class.toString(Class.java:126) at java.lang.String.valueOf(String.java:2615) at java.lang.StringBuilder.append(StringBuilder.java:116) at com.sun.org.apache.xml.internal.dtm.ObjectFactory.lookUpFactoryClass(ObjectFactory.java:220) at com.sun.org.apache.xml.internal.dtm.ObjectFactory.createObject(ObjectFactory.java:131) at com.sun.org.apache.xml.internal.dtm.ObjectFactory.createObject(ObjectFactory.java:101) at com.sun.org.apache.xml.internal.dtm.DTMManager.newInstance(DTMManager.java:135) at com.sun.org.apache.xpath.internal.XPathContext.<init>(XPathContext.java:87) at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.eval(XPathImpl.java:201) at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:275) at org.milyn.xml.XmlUtil.getNodeList(XmlUtil.java:366) at org.milyn.xml.XmlUtil.getNode(XmlUtil.java:384) at org.milyn.cdr.XMLConfigDigester.digestResourceConfig(XMLConfigDigester.java:205) at org.milyn.cdr.XMLConfigDigester.digestV10XSDValidatedConfig(XMLConfigDigester.java:171) at org.milyn.cdr.XMLConfigDigester.digestConfig(XMLConfigDigester.java:93) at org.milyn.cdr.XMLConfigDigester.digestConfig(XMLConfigDigester.java:68) at org.milyn.cdr.SmooksResourceConfigurationStore.registerResources(SmooksResourceConfigurationStore.java:230) at org.milyn.cdr.SmooksResourceConfigurationStore.registerInstalledResources(SmooksResourceConfigurationStore.java:167) at org.milyn.cdr.SmooksResourceConfigurationStore.<init>(SmooksResourceConfigurationStore.java:104) at org.milyn.container.standalone.StandaloneApplicationContext.<init>(StandaloneApplicationContext.java:51) at org.milyn.Smooks.<init>(Smooks.java:93) at org.milyn.Smooks.<init>(Smooks.java:129) at org.jboss.soa.esb.services.security.auth.ws.WSSecurityInfoExtractor.createSmooks(WSSecurityInfoExtractor.java:124) at org.jboss.soa.esb.services.security.auth.ws.WSSecurityInfoExtractor.<init>(WSSecurityInfoExtractor.java:67) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener$JBossRemotingMessageComposer.populateMessage(JBossRemotingGatewayListener.java:523) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener$JBossRemotingMessageComposer.populateMessage(JBossRemotingGatewayListener.java:487) at org.jboss.soa.esb.listeners.message.AbstractMessageComposer.compose(AbstractMessageComposer.java:76) at org.jboss.soa.esb.listeners.message.UncomposedMessageDeliveryAdapter.deliverSyncWithoutDecomposing(UncomposedMessageDeliveryAdapter.java:104) at org.jboss.soa.esb.listeners.message.UncomposedMessageDeliveryAdapter.deliverSync(UncomposedMessageDeliveryAdapter.java:86) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener.invoke(JBossRemotingGatewayListener.java:345) 14:01:03,547 ERROR [JBossRemotingGatewayListener] JBoss Remoting Gateway failed to synchronously deliver message to target service [WSProxy:WSProxy]. java.lang.OutOfMemoryError: Java heap space at com.sun.org.apache.xml.internal.utils.ObjectVector.<init>(ObjectVector.java:68) at com.sun.org.apache.xml.internal.utils.ObjectStack.<init>(ObjectStack.java:51) at com.sun.org.apache.xpath.internal.XPathContext.<init>(XPathContext.java:856) at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.eval(XPathImpl.java:201) at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:275) at org.milyn.xml.XmlUtil.getNodeList(XmlUtil.java:366) at org.milyn.xml.XmlUtil.getNode(XmlUtil.java:384) at org.milyn.cdr.XMLConfigDigester.digestParameters(XMLConfigDigester.java:287) at org.milyn.cdr.XMLConfigDigester.digestResourceConfig(XMLConfigDigester.java:245) at org.milyn.cdr.XMLConfigDigester.digestV10XSDValidatedConfig(XMLConfigDigester.java:171) at org.milyn.cdr.XMLConfigDigester.digestConfig(XMLConfigDigester.java:93) at org.milyn.cdr.XMLConfigDigester.digestConfig(XMLConfigDigester.java:68) at org.milyn.cdr.SmooksResourceConfigurationStore.registerResources(SmooksResourceConfigurationStore.java:230) at org.milyn.Smooks.addConfigurations(Smooks.java:182) at org.milyn.Smooks.addConfigurations(Smooks.java:200) at org.milyn.Smooks.<init>(Smooks.java:130) at org.jboss.soa.esb.services.security.auth.ws.WSSecurityInfoExtractor.createSmooks(WSSecurityInfoExtractor.java:124) at org.jboss.soa.esb.services.security.auth.ws.WSSecurityInfoExtractor.<init>(WSSecurityInfoExtractor.java:67) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener$JBossRemotingMessageComposer.populateMessage(JBossRemotingGatewayListener.java:523) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener$JBossRemotingMessageComposer.populateMessage(JBossRemotingGatewayListener.java:487) at org.jboss.soa.esb.listeners.message.AbstractMessageComposer.compose(AbstractMessageComposer.java:76) at org.jboss.soa.esb.listeners.message.UncomposedMessageDeliveryAdapter.deliverSyncWithoutDecomposing(UncomposedMessageDeliveryAdapter.java:104) at org.jboss.soa.esb.listeners.message.UncomposedMessageDeliveryAdapter.deliverSync(UncomposedMessageDeliveryAdapter.java:86) at org.jboss.soa.esb.listeners.gateway.JBossRemotingGatewayListener.invoke(JBossRemotingGatewayListener.java:345) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:866) at org.jboss.remoting.transport.coyote.CoyoteInvoker.service(CoyoteInvoker.java:310) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:595) 14:01:23,159 WARN [AbstractThreadedManagedLifecycle] Unexpected error from doRun() java.lang.OutOfMemoryError: Java heap space -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira