[jboss-jira] [JBoss JIRA] (AS7-3796) Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES (ChannelBusyException: Too many open outbound writes)
Radoslav Husar (JIRA)
jira-events at lists.jboss.org
Thu Feb 16 11:47:36 EST 2012
[ https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667152#comment-12667152 ]
Radoslav Husar commented on AS7-3796:
-------------------------------------
After I upgraded xnio-nio to correct version another suspicious fact is that the throughput didn't scale with increasing clients.
This might be easily related to clustering issue or network instability in the lab that we have been seeing past weeks.
Now
{noformat}
Nodes: 2, Sessions: 200, active: 200, samples: 170168, throughput 1,416.7 samples/s, 0.0 MB/s, mean response: 40 ms, sampling errors: 0, invalid samples: 0, valid samples: 170168 (100%)
Nodes: 2, Sessions: 400, active: 400, samples: 172565, throughput 1,437.0 samples/s, 0.0 MB/s, mean response: 177 ms, sampling errors: 0, invalid samples: 0, valid samples: 172565 (100%)
Nodes: 2, Sessions: 600, active: 600, samples: 169105, throughput 1,408.0 samples/s, 0.0 MB/s, mean response: 324 ms, sampling errors: 0, invalid samples: 0, valid samples: 169105 (100%)
Nodes: 2, Sessions: 800, active: 800, samples: 171483, throughput 1,427.4 samples/s, 0.0 MB/s, mean response: 459 ms, sampling errors: 0, invalid samples: 0, valid samples: 171483 (100%)
Nodes: 2, Sessions: 1000, active: 1000, samples: 173309, throughput 1,442.8 samples/s, 0.0 MB/s, mean response: 592 ms, sampling errors: 0, invalid samples: 0, valid samples: 173309 (100%)
Nodes: 2, Sessions: 1200, active: 1200, samples: 172633, throughput 1,437.4 samples/s, 0.0 MB/s, mean response: 733 ms, sampling errors: 0, invalid samples: 0, valid samples: 172633 (100%)
Nodes: 2, Sessions: 1400, active: 1400, samples: 164112, throughput 1,366.2 samples/s, 0.0 MB/s, mean response: 923 ms, sampling errors: 0, invalid samples: 0, valid samples: 164112 (100%)
Nodes: 2, Sessions: 1600, active: 1600, samples: 166286, throughput 1,384.2 samples/s, 0.0 MB/s, mean response: 1054 ms, sampling errors: 0, invalid samples: 0, valid samples: 166286 (100%)
Nodes: 2, Sessions: 1800, active: 1800, samples: 170744, throughput 1,421.3 samples/s, 0.0 MB/s, mean response: 1165 ms, sampling errors: 0, invalid samples: 0, valid samples: 170744 (100%)
Nodes: 2, Sessions: 2000, active: 2000, samples: 174106, throughput 1,449.2 samples/s, 0.0 MB/s, mean response: 1279 ms, sampling errors: 0, invalid samples: 0, valid samples: 174106 (100%)
Nodes: 2, Sessions: 2200, active: 2200, samples: 174016, throughput 1,448.7 samples/s, 0.0 MB/s, mean response: 1415 ms, sampling errors: 1, invalid samples: 0, valid samples: 174015 (99%)
{noformat}
Before
{noformat}
Nodes: 2, Sessions: 200, active: 200, samples: 230811, throughput 1,923.4 samples/s, 0.0 MB/s, mean response: 3 ms, sampling errors: 0, invalid samples: 0, valid samples: 230811 (100%)
Nodes: 2, Sessions: 400, active: 400, samples: 451720, throughput 3,764.2 samples/s, 0.0 MB/s, mean response: 5 ms, sampling errors: 0, invalid samples: 0, valid samples: 451720 (100%)
Nodes: 2, Sessions: 600, active: 600, samples: 590932, throughput 4,924.3 samples/s, 0.0 MB/s, mean response: 20 ms, sampling errors: 0, invalid samples: 0, valid samples: 590932 (100%)
Nodes: 2, Sessions: 800, active: 800, samples: 606093, throughput 5,050.6 samples/s, 0.0 MB/s, mean response: 56 ms, sampling errors: 0, invalid samples: 0, valid samples: 606093 (100%)
Nodes: 2, Sessions: 1000, active: 1000, samples: 608173, throughput 5,068.0 samples/s, 0.0 MB/s, mean response: 95 ms, sampling errors: 0, invalid samples: 0, valid samples: 608173 (100%)
Nodes: 2, Sessions: 1200, active: 1200, samples: 625177, throughput 5,209.7 samples/s, 0.0 MB/s, mean response: 128 ms, sampling errors: 0, invalid samples: 0, valid samples: 625177 (100%)
Nodes: 2, Sessions: 1400, active: 1400, samples: 604213, throughput 5,034.9 samples/s, 0.0 MB/s, mean response: 175 ms, sampling errors: 282, invalid samples: 0, valid samples: 603931 (99%)
{noformat}
> Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES (ChannelBusyException: Too many open outbound writes)
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: AS7-3796
> URL: https://issues.jboss.org/browse/AS7-3796
> Project: Application Server 7
> Issue Type: Bug
> Components: Clustering, EJB, Remoting
> Affects Versions: 7.1.0.Final
> Reporter: Radoslav Husar
> Assignee: David Lloyd
> Priority: Critical
> Labels: eap6_LA
> Fix For: 7.1.1.Final
>
>
> David and Jaikiran,
> lets revisit this topic. I still think were are leaking messages in remoting3. Badly. Even though I set MAX_O_Ms to 100K it still results in:
> {noformat}
> 2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid response. Runner: 227.>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid response. Runner: 227.
> at org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at $Proxy0.getSerialAndIncrement(Unknown Source)
> at org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
> ... 3 more
> Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
> at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
> at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
> at org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
> at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
> at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
> at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
> at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
> at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
> at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
> at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
> at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
> ... 5 more
> {noformat}
> Here is a snip from the client properties:
> {noformat}
> remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
> remote.connections=perf18,perf19,perf20,perf21
> remote.connection.perf18.host=perf18
> remote.connection.perf18.port=4447
> remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
> remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list