[JBoss JIRA] (AS7-3796) Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES (ChannelBusyException: Too many open outbound writes)
8:08 a.m.
Radoslav Husar created AS7-3796:
-----------------------------------
Summary: Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly. Even
though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:10 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
Radoslav Husar commented on AS7-3796:
-------------------------------------
I should mention that this happens easily with 400 clients each doing requests
sequentially with a delay in between, so there is no way there could be more than 100000
in the queue already.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:08 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
David Lloyd commented on AS7-3796:
----------------------------------
The only way to definitively prove a leak is to execute a small number of invocations (say
less than 10 at a time) repeatedly. If the exception is thrown, it's a leak,
otherwise no.
Also, make *sure* your client is using the latest Remoting, XNIO, and EJB Client JARs.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:12 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
jaikiran pai commented on AS7-3796:
-----------------------------------
I have a test which tries to do something similar in AS7. I had run that while
implementing this solution. I'll rerun it again and send a pull request for upstream
so that we have this thoroughly tested. In the meantime, I'll see if we are missing
some path which might not be using the semaphore before writing out to the channel(I
don't think we have missed any, but will check again).
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:20 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
jaikiran pai commented on AS7-3796:
-----------------------------------
Here's the stress test I was talking about
https://github.com/jaikiran/jboss-as/commit/d53c38511958a444032732e9e6eeb....
Specifically this test
https://github.com/jaikiran/jboss-as/blob/d53c38511958a444032732e9e6eebd9...
triggers 400 threads to do 50 invocations each simultaneously. Note that I haven't
changed any default values of the MAX_OUTBOUND_MESSAGE count, which means the default
value which is as low as 80 is being used. The test is passing.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:47 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
Radoslav Husar commented on AS7-3796:
-------------------------------------
After I upgraded xnio-nio to correct version another suspicious fact is that the
throughput didn't scale with increasing clients.
This might be easily related to clustering issue or network instability in the lab that we
have been seeing past weeks.
Now
{noformat}
Nodes: 2, Sessions: 200, active: 200, samples: 170168, throughput 1,416.7 samples/s, 0.0
MB/s, mean response: 40 ms, sampling errors: 0, invalid samples: 0, valid samples: 170168
(100%)
Nodes: 2, Sessions: 400, active: 400, samples: 172565, throughput 1,437.0 samples/s, 0.0
MB/s, mean response: 177 ms, sampling errors: 0, invalid samples: 0, valid samples: 172565
(100%)
Nodes: 2, Sessions: 600, active: 600, samples: 169105, throughput 1,408.0 samples/s, 0.0
MB/s, mean response: 324 ms, sampling errors: 0, invalid samples: 0, valid samples: 169105
(100%)
Nodes: 2, Sessions: 800, active: 800, samples: 171483, throughput 1,427.4 samples/s, 0.0
MB/s, mean response: 459 ms, sampling errors: 0, invalid samples: 0, valid samples: 171483
(100%)
Nodes: 2, Sessions: 1000, active: 1000, samples: 173309, throughput 1,442.8 samples/s, 0.0
MB/s, mean response: 592 ms, sampling errors: 0, invalid samples: 0, valid samples: 173309
(100%)
Nodes: 2, Sessions: 1200, active: 1200, samples: 172633, throughput 1,437.4 samples/s, 0.0
MB/s, mean response: 733 ms, sampling errors: 0, invalid samples: 0, valid samples: 172633
(100%)
Nodes: 2, Sessions: 1400, active: 1400, samples: 164112, throughput 1,366.2 samples/s, 0.0
MB/s, mean response: 923 ms, sampling errors: 0, invalid samples: 0, valid samples: 164112
(100%)
Nodes: 2, Sessions: 1600, active: 1600, samples: 166286, throughput 1,384.2 samples/s, 0.0
MB/s, mean response: 1054 ms, sampling errors: 0, invalid samples: 0, valid samples:
166286 (100%)
Nodes: 2, Sessions: 1800, active: 1800, samples: 170744, throughput 1,421.3 samples/s, 0.0
MB/s, mean response: 1165 ms, sampling errors: 0, invalid samples: 0, valid samples:
170744 (100%)
Nodes: 2, Sessions: 2000, active: 2000, samples: 174106, throughput 1,449.2 samples/s, 0.0
MB/s, mean response: 1279 ms, sampling errors: 0, invalid samples: 0, valid samples:
174106 (100%)
Nodes: 2, Sessions: 2200, active: 2200, samples: 174016, throughput 1,448.7 samples/s, 0.0
MB/s, mean response: 1415 ms, sampling errors: 1, invalid samples: 0, valid samples:
174015 (99%)
{noformat}
Before
{noformat}
Nodes: 2, Sessions: 200, active: 200, samples: 230811, throughput 1,923.4 samples/s, 0.0
MB/s, mean response: 3 ms, sampling errors: 0, invalid samples: 0, valid samples: 230811
(100%)
Nodes: 2, Sessions: 400, active: 400, samples: 451720, throughput 3,764.2 samples/s, 0.0
MB/s, mean response: 5 ms, sampling errors: 0, invalid samples: 0, valid samples: 451720
(100%)
Nodes: 2, Sessions: 600, active: 600, samples: 590932, throughput 4,924.3 samples/s, 0.0
MB/s, mean response: 20 ms, sampling errors: 0, invalid samples: 0, valid samples: 590932
(100%)
Nodes: 2, Sessions: 800, active: 800, samples: 606093, throughput 5,050.6 samples/s, 0.0
MB/s, mean response: 56 ms, sampling errors: 0, invalid samples: 0, valid samples: 606093
(100%)
Nodes: 2, Sessions: 1000, active: 1000, samples: 608173, throughput 5,068.0 samples/s, 0.0
MB/s, mean response: 95 ms, sampling errors: 0, invalid samples: 0, valid samples: 608173
(100%)
Nodes: 2, Sessions: 1200, active: 1200, samples: 625177, throughput 5,209.7 samples/s, 0.0
MB/s, mean response: 128 ms, sampling errors: 0, invalid samples: 0, valid samples: 625177
(100%)
Nodes: 2, Sessions: 1400, active: 1400, samples: 604213, throughput 5,034.9 samples/s, 0.0
MB/s, mean response: 175 ms, sampling errors: 282, invalid samples: 0, valid samples:
603931 (99%)
{noformat}
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:47 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
Radoslav Husar updated AS7-3796:
--------------------------------
Comment: was deleted
(was: After I upgraded xnio-nio to correct version another suspicious fact is that the
throughput didn't scale with increasing clients.
This might be easily related to clustering issue or network instability in the lab that we
have been seeing past weeks.
Now
{noformat}
Nodes: 2, Sessions: 200, active: 200, samples: 170168, throughput 1,416.7 samples/s, 0.0
MB/s, mean response: 40 ms, sampling errors: 0, invalid samples: 0, valid samples: 170168
(100%)
Nodes: 2, Sessions: 400, active: 400, samples: 172565, throughput 1,437.0 samples/s, 0.0
MB/s, mean response: 177 ms, sampling errors: 0, invalid samples: 0, valid samples: 172565
(100%)
Nodes: 2, Sessions: 600, active: 600, samples: 169105, throughput 1,408.0 samples/s, 0.0
MB/s, mean response: 324 ms, sampling errors: 0, invalid samples: 0, valid samples: 169105
(100%)
Nodes: 2, Sessions: 800, active: 800, samples: 171483, throughput 1,427.4 samples/s, 0.0
MB/s, mean response: 459 ms, sampling errors: 0, invalid samples: 0, valid samples: 171483
(100%)
Nodes: 2, Sessions: 1000, active: 1000, samples: 173309, throughput 1,442.8 samples/s, 0.0
MB/s, mean response: 592 ms, sampling errors: 0, invalid samples: 0, valid samples: 173309
(100%)
Nodes: 2, Sessions: 1200, active: 1200, samples: 172633, throughput 1,437.4 samples/s, 0.0
MB/s, mean response: 733 ms, sampling errors: 0, invalid samples: 0, valid samples: 172633
(100%)
Nodes: 2, Sessions: 1400, active: 1400, samples: 164112, throughput 1,366.2 samples/s, 0.0
MB/s, mean response: 923 ms, sampling errors: 0, invalid samples: 0, valid samples: 164112
(100%)
Nodes: 2, Sessions: 1600, active: 1600, samples: 166286, throughput 1,384.2 samples/s, 0.0
MB/s, mean response: 1054 ms, sampling errors: 0, invalid samples: 0, valid samples:
166286 (100%)
Nodes: 2, Sessions: 1800, active: 1800, samples: 170744, throughput 1,421.3 samples/s, 0.0
MB/s, mean response: 1165 ms, sampling errors: 0, invalid samples: 0, valid samples:
170744 (100%)
Nodes: 2, Sessions: 2000, active: 2000, samples: 174106, throughput 1,449.2 samples/s, 0.0
MB/s, mean response: 1279 ms, sampling errors: 0, invalid samples: 0, valid samples:
174106 (100%)
Nodes: 2, Sessions: 2200, active: 2200, samples: 174016, throughput 1,448.7 samples/s, 0.0
MB/s, mean response: 1415 ms, sampling errors: 1, invalid samples: 0, valid samples:
174015 (99%)
{noformat}
Before
{noformat}
Nodes: 2, Sessions: 200, active: 200, samples: 230811, throughput 1,923.4 samples/s, 0.0
MB/s, mean response: 3 ms, sampling errors: 0, invalid samples: 0, valid samples: 230811
(100%)
Nodes: 2, Sessions: 400, active: 400, samples: 451720, throughput 3,764.2 samples/s, 0.0
MB/s, mean response: 5 ms, sampling errors: 0, invalid samples: 0, valid samples: 451720
(100%)
Nodes: 2, Sessions: 600, active: 600, samples: 590932, throughput 4,924.3 samples/s, 0.0
MB/s, mean response: 20 ms, sampling errors: 0, invalid samples: 0, valid samples: 590932
(100%)
Nodes: 2, Sessions: 800, active: 800, samples: 606093, throughput 5,050.6 samples/s, 0.0
MB/s, mean response: 56 ms, sampling errors: 0, invalid samples: 0, valid samples: 606093
(100%)
Nodes: 2, Sessions: 1000, active: 1000, samples: 608173, throughput 5,068.0 samples/s, 0.0
MB/s, mean response: 95 ms, sampling errors: 0, invalid samples: 0, valid samples: 608173
(100%)
Nodes: 2, Sessions: 1200, active: 1200, samples: 625177, throughput 5,209.7 samples/s, 0.0
MB/s, mean response: 128 ms, sampling errors: 0, invalid samples: 0, valid samples: 625177
(100%)
Nodes: 2, Sessions: 1400, active: 1400, samples: 604213, throughput 5,034.9 samples/s, 0.0
MB/s, mean response: 175 ms, sampling errors: 282, invalid samples: 0, valid samples:
603931 (99%)
{noformat}
)
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:01 p.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
jaikiran pai commented on AS7-3796:
-----------------------------------
Here's another test that I added
https://github.com/jaikiran/jboss-as/blob/78114be3188534df7f587f32a0d9644...
which is similar to previous one but this one has few threads but being invoked repeatedly
multiple times. This one passes too (tried more than once).
I'm going to take a more detailed look at the performance setup, tomorrow, and see if
I can find something amiss.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:12 p.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
Dimitris Andreadis updated AS7-3796:
------------------------------------
In order to cherrypick this in the EAP ER2 build, we would need a resolution by Feb/27th.
Is this doable?
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:16 p.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
David Lloyd commented on AS7-3796:
----------------------------------
Only if we actually duplicate the problem.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:28 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
jaikiran pai commented on AS7-3796:
-----------------------------------
I'm preparing a couple of jars for the EJB client and the remoting project to add some
more detailed logging (using a different category) for the purpose of getting more
information when this exception occurs on the perf setup. Let's retest this on the
perf environment using those jars.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:14 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
jaikiran pai commented on AS7-3796:
-----------------------------------
I've deployed the following 2 snapshots to Maven repository:
1.0.3.Final-SNAPSHOT here
https://repository.jboss.org/nexus/content/repositories/snapshots/org/jbo...
3.2.3.GA-SNAPSHOT here
https://repository.jboss.org/nexus/content/repositories/snapshots/org/jbo...
Both these jars mainly have logging changes to introduce more logging. Please enable INFO
level logging for the following packages *and redirect these packages to a common log
file* (but don't let them log into the file which all other packages use):
For remoting the package is - remoting.perf.debug
For EJB client it is - ejb.client.perf.debug
I don't know how the perf setup is built, so even though these are client side
changes, it might make sense to build even the AS7 using these jars.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:09 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
jaikiran pai commented on AS7-3796:
-----------------------------------
We release a new version of jboss-ejb-client today. I've uploaded a new snapshot
containing some fixes (which aren't related to this issue) and these additional
logging messages. Please use the 1.0.4.Final-SNAPSHOT of jboss-ejb-client which is
available here
https://repository.jboss.org/nexus/content/repositories/snapshots/org/jbo....
Rest of the instructions are the same as in my previous reply.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:09 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
jaikiran pai edited comment on AS7-3796 at 2/27/12 6:09 AM:
------------------------------------------------------------
We released a new version of jboss-ejb-client today. I've uploaded a new snapshot
containing some fixes (which aren't related to this issue) and these additional
logging messages. Please use the 1.0.4.Final-SNAPSHOT of jboss-ejb-client which is
available here
https://repository.jboss.org/nexus/content/repositories/snapshots/org/jbo....
Rest of the instructions are the same as in my previous reply.
was (Author: jaikiran):
We release a new version of jboss-ejb-client today. I've uploaded a new snapshot
containing some fixes (which aren't related to this issue) and these additional
logging messages. Please use the 1.0.4.Final-SNAPSHOT of jboss-ejb-client which is
available here
https://repository.jboss.org/nexus/content/repositories/snapshots/org/jbo....
Rest of the instructions are the same as in my previous reply.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:43 p.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
jaikiran pai commented on AS7-3796:
-----------------------------------
Rado provided the logs from build#28. Those do not contain any ChannelBusyException(s).
We'll have to try and reproduce this to be able to fix this. I'll be watching the
next few runs and check with Rado if he runs into this again.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:29 p.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
David Lloyd resolved AS7-3796.
------------------------------
Resolution: Done
I'm calling this resolved based on the evidence.
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:39 a.m.
[
https://issues.jboss.org/browse/AS7-3796?page=com.atlassian.jira.plugin.s...
]
Radoslav Husar closed AS7-3796.
-------------------------------
Remote EJB client seems to be leaking messages MAX_OUTBOUND_MESSAGES
(ChannelBusyException: Too many open outbound writes)
--------------------------------------------------------------------------------------------------------------------------
Key: AS7-3796
URL: https://issues.jboss.org/browse/AS7-3796
Project: Application Server 7
Issue Type: Bug
Components: Clustering, EJB, Remoting
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: David Lloyd
Priority: Critical
Labels: eap6_LA
Fix For: 7.1.1.Final
David and Jaikiran,
lets revisit this topic. I still think were are leaking messages in remoting3. Badly.
Even though I set MAX_O_Ms to 100K it still results in:
{noformat}
2012/02/16 08:34:48:728 EST [WARN ][Runner - 227] SFCORE_LOG - Error sampling data:
<org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Could not get valid
response. Runner: 227.
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:79)
at
org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getSerialAndIncrement(Unknown Source)
at
org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImpl.java:68)
... 3 more
Caused by: org.jboss.remoting3.ChannelBusyException: Too many open outbound writes
at
org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:111)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296)
at
org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:235)
at
org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:193)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:173)
at
org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:92)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:175)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
... 5 more
{noformat}
Here is a snip from the client properties:
{noformat}
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=perf18,perf19,perf20,perf21
remote.connection.perf18.host=perf18
remote.connection.perf18.port=4447
remote.connection.perf18.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.perf18.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES=100000
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4419
days inactive
4697
days old
16 comments
4 participants
participants (4)
-
David Lloyd (JIRA) -
Dimitris Andreadis (JIRA)
-
jaikiran pai (JIRA)
-
Radoslav Husar (JIRA)