[jboss-user] [JBoss Remoting] - SSL bisocket socketWrite0() hangs, while holding Messaging Post Office lock (on a network failure)

Todd Sandor do-not-reply at jboss.com
Fri Aug 17 17:31:34 EDT 2012 

Todd Sandor [https://community.jboss.org/people/toddler] created the discussion

"SSL bisocket socketWrite0() hangs, while holding Messaging Post Office lock (on a network failure)"

To view the discussion, visit: https://community.jboss.org/message/754327#754327

--------------------------------------------------------------
Hi ...

We are using EAP5.1.1 (remoting-2-4-5.SP2) and experience a problem when using
SSL and there is a network hard failure (e.g. ethernet cable pulled), a socketWrite0() hangs
while holding a  Messaging Post Office lock (readLock()) - this hangs JMS...(many threads
need PostOffice).  Thread dump below.  It is difficult to reproduce.

A Solaris 10 environment.

This problem has only been observed when SSL is used.

In our remoting-bisocket-service.xml file, we use:
 <attribute isParam="true" name="writeTimeout">30000</attribute>

Have been told "writeTimeout" is broken when SSL is being used.

It has been suggested, for SSL, we change the configuration to turn on the check_connection parameter:
<attribute isParam="true" name="socket.check_connection">true</attribute>

But my concern is:
* results in an extra round-trip I/O (not good for a variety of reasons - throughput, remote clients with high latency connecitons, etc.).
* potential to hit other problems with check_connections enabled.   e.g. found this:
 https://community.jboss.org/message/369442#369442 https://community.jboss.org/message/369442

Has anyone experienced this problem and/or know a solution (other then using
"socket.check_connection") might be?
Searched and found a number of remoting issues on hard-network failure, but not this particular one...

Potentially Related Issue:
 https://issues.jboss.org/browse/JBREM-1307 https://issues.jboss.org/browse/JBREM-1307

Thead Dump below:

2012.07.26 13:23:38 -0500 INFO  [STDOUT] "WorkerThread#1[10.143.109.23:60321]" Id=74 RUNNABLE (in native)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:297)
    at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:286)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:743)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:731)
    at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
    -  locked com.sun.net.ssl.internal.ssl.AppOutputStream at ac9224
    at org.jboss.remoting.transport.socket.TimedOutputStream.write(TimedOutputStream.java:119)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
    -  locked java.io.BufferedOutputStream at 1aea5f0
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    -  locked java.io.DataOutputStream at 16e88df
    at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
    at org.jboss.messaging.core.impl.message.MessageSupport.write(MessageSupport.java:387)
    at org.jboss.jms.message.JBossMessage.write(JBossMessage.java:1069)
    at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:91)
    at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:1325)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:861)
    at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:470)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:169)
    at org.jboss.remoting.Client.invoke(Client.java:2070)
    at org.jboss.remoting.Client.invoke(Client.java:879)
    at org.jboss.remoting.Client.invokeOneway(Client.java:928)
    at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:835)
    at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:708)
    at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1600)
    at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1512)
    -  locked org.jboss.jms.server.endpoint.ServerSessionEndpoint at 1105e96
    at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:353)
    -  locked java.lang.Object at 1b1dde3
    at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
    at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:617)
    at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
    at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:681)
    at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:540)
    at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:251)
    -  locked java.lang.Object at 5e7463
    at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:3133)
    at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:956)
    at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:794)
    at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:435)
    at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
    at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
    at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:158)
    at sun.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at com.timetra.nms.server.j2ee.jms.messaging.JBossMessagingServerInterceptor.handleSessionSend(JBossMessagingServerInterceptor.java:346)
    at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
    at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
    at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
    at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
    at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
    -  locked org.jboss.remoting.transport.socket.ServerThread at 465cc
    at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
    at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

    Number of locked synchronizers = 1
    - java.util.concurrent.locks.ReentrantLock$NonfairSync at 12b4a3e
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/754327#754327]

Start a new discussion in JBoss Remoting at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2050]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20120817/9eeaa147/attachment-0001.html

More information about the jboss-user mailing list