4:31 p.m.
Todd Sandor [https://community.jboss.org/people/toddler] created the discussion
"SSL bisocket socketWrite0() hangs, while holding Messaging Post Office lock (on a
network failure)"
To view the discussion, visit: https://community.jboss.org/message/754327#754327
--------------------------------------------------------------
Hi ...
We are using EAP5.1.1 (remoting-2-4-5.SP2) and experience a problem when using
SSL and there is a network hard failure (e.g. ethernet cable pulled), a socketWrite0()
hangs
while holding a Messaging Post Office lock (readLock()) - this hangs JMS...(many threads
need PostOffice). Thread dump below. It is difficult to reproduce.
A Solaris 10 environment.
This problem has only been observed when SSL is used.
In our remoting-bisocket-service.xml file, we use:
<attribute isParam="true"
name="writeTimeout">30000</attribute>
Have been told "writeTimeout" is broken when SSL is being used.
It has been suggested, for SSL, we change the configuration to turn on the
check_connection parameter:
<attribute isParam="true"
name="socket.check_connection">true</attribute>
But my concern is:
* results in an extra round-trip I/O (not good for a variety of reasons - throughput,
remote clients with high latency connecitons, etc.).
* potential to hit other problems with check_connections enabled. e.g. found this:
https://community.jboss.org/message/369442#369442
https://community.jboss.org/message/369442
Has anyone experienced this problem and/or know a solution (other then using
"socket.check_connection") might be?
Searched and found a number of remoting issues on hard-network failure, but not this
particular one...
Potentially Related Issue:
https://issues.jboss.org/browse/JBREM-1307 https://issues.jboss.org/browse/JBREM-1307
Thead Dump below:
2012.07.26 13:23:38 -0500 INFO [STDOUT] "WorkerThread#1[10.143.109.23:60321]"
Id=74 RUNNABLE (in native)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:297)
at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:286)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:743)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:731)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
- locked com.sun.net.ssl.internal.ssl.AppOutputStream@ac9224
at
org.jboss.remoting.transport.socket.TimedOutputStream.write(TimedOutputStream.java:119)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
- locked java.io.BufferedOutputStream@1aea5f0
at java.io.DataOutputStream.write(DataOutputStream.java:90)
- locked java.io.DataOutputStream@16e88df
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at
org.jboss.messaging.core.impl.message.MessageSupport.write(MessageSupport.java:387)
at org.jboss.jms.message.JBossMessage.write(JBossMessage.java:1069)
at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:91)
at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
at
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:1325)
at
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:861)
at
org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:470)
at
org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:169)
at org.jboss.remoting.Client.invoke(Client.java:2070)
at org.jboss.remoting.Client.invoke(Client.java:879)
at org.jboss.remoting.Client.invokeOneway(Client.java:928)
at
org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:835)
at
org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:708)
at
org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1600)
at
org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1512)
- locked org.jboss.jms.server.endpoint.ServerSessionEndpoint@1105e96
at
org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:353)
- locked java.lang.Object@1b1dde3
at
org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
at
org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:617)
at
org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
at
org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:681)
at
org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:540)
at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:251)
- locked java.lang.Object@5e7463
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:3133)
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:956)
at
org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:794)
at
org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:435)
at
org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
at
org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:158)
at sun.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at
com.timetra.nms.server.j2ee.jms.messaging.JBossMessagingServerInterceptor.handleSessionSend(JBossMessagingServerInterceptor.java:346)
at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
at
org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
at
org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
at
org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
- locked org.jboss.remoting.transport.socket.ServerThread@465cc
at
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync@12b4a3e
--------------------------------------------------------------
Reply to this message by going to Community
[https://community.jboss.org/message/754327#754327]
Start a new discussion in JBoss Remoting at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&...]