[jboss-user] [JBoss Messaging] - Re: JBM deadlock cause by socket write
lanceliao1
do-not-reply at jboss.com
Mon Nov 24 02:57:26 EST 2008
Socket info
Solaris: netstat -an
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------ -----------
192.166.32.8.4557 192.166.32.155.52066 64679 0 49640 0 ESTABLISHED
192.166.32.8.4557 192.166.32.180.3199 65535 0 49640 0 ESTABLISHED
192.166.32.8.4557 192.166.32.96.1601 65535 0 0 0 ESTABLISHED
192.166.32.8.56726 192.166.32.8.4557 49152 0 49152 0 ESTABLISHED
192.166.32.8.4557 192.166.32.244.4245 65535 0 49640 0 ESTABLISHED
192.166.32.8.4557 192.166.32.155.52297 64606 0 49640 0 ESTABLISHED
192.166.32.8.4557 192.166.88.88.3746 65520 0 49640 0 ESTABLISHED
192.166.32.8.4557 192.166.88.88.1593 64647 0 49640 0 ESTABLISHED
192.166.32.8.4557 192.166.32.21.1872 65535 0 0 0 ESTABLISHED
192.166.32.8.4557 192.166.32.100.7408 64196 0 49640 0 ESTABLISHED
192.166.32.8.4557 192.166.32.180.2108 64149 0 49640 0 ESTABLISHED
192.166.32.8.4557 192.166.32.21.2152 65535 0 0 0 ESTABLISHED
192.166.32.8.4557 192.166.32.21.2154 65535 0 0 0 ESTABLISHED
... more than 200 line.
The following lines Rwind are zero,meaning the socketwrite() kept waiting(more than 2 hours,never return),the correspond reader thread does not working.
192.166.32.8.4557 192.166.32.96.1601
192.166.32.8.4557 192.166.32.21.1872
192.166.32.8.4557 192.166.32.21.2152
192.166.32.8.4557 192.166.32.21.2154
***************************************************************************************************************************
Thread dump(full dump are too large,only select some)
| "WorkerThread#47[192.166.32.96:1601]" prio=3 tid=0x0139eb68 nid=0x264 in Object.wait() [0x4be7f000..0x4be7fa70]
| at java.lang.Object.wait(Native Method)
| at java.lang.Object.wait(Object.java:474)
| at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
| - locked <0x7964b110> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
| at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
| at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
| at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
| at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
| at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
| at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
| at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
| at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
|
| "WorkerThread#36[192.166.32.21:1872]" prio=3 tid=0x00fc1a00 nid=0x23f in Object.wait() [0x4de7f000..0x4de7faf0]
| at java.lang.Object.wait(Native Method)
| at java.lang.Object.wait(Object.java:474)
| at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
| - locked <0x7964b110> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
| at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
| at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
| at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
| at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
| at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
| at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
| at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
| at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
|
| "WorkerThread#164[192.166.32.21:2152]" prio=3 tid=0x02862648 nid=0x4d22 in Object.wait() [0x4897f000..0x4897f970]
| at java.lang.Object.wait(Native Method)
| at java.lang.Object.wait(Object.java:474)
| at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
| - locked <0x7964b110> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
| at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
| at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
| at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
| at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
| at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
| at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
| at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
| at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
|
| "WorkerThread#165[192.166.32.21:2154]" prio=3 tid=0x0383c480 nid=0x4d24 in Object.wait() [0x4947f000..0x4947fa70]
| at java.lang.Object.wait(Native Method)
| at java.lang.Object.wait(Object.java:474)
| at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
| - locked <0x7964b110> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
| at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
| at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
| at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
| at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
| at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
| at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
| at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
| at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
|
********************************************************************************************************************************
All these thread are waiting on MessagingPostOffice.routInternal(2075),waiting a readwrite lock.
lock.readLock().acquire();
the lock holder is:
| "WorkerThread#159[192.166.32.8:33676]" prio=3 tid=0x0242f938 nid=0x4cdd runnable [0x4667e000..0x4667f9f0]
| at java.net.SocketOutputStream.socketWrite0(Native Method)
| at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
| at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
| at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
| at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
| - locked <0x7af9d5c0> (a java.io.BufferedOutputStream)
| at java.io.DataOutputStream.flush(DataOutputStream.java:106)
| at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
| at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
| at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:945)
| at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:586)
| at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
| at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
| at org.jboss.remoting.Client.invoke(Client.java:1634)
| at org.jboss.remoting.Client.invoke(Client.java:548)
| at org.jboss.remoting.Client.invokeOneway(Client.java:598)
| at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
| at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
| - locked <0x794462d8> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
| at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
| - locked <0x7a778760> (a java.lang.Object)
| at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
| at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
| at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
| at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
| at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
| at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
| - locked <0x796c3d18> (a java.lang.Object)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
| at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
| at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
| at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
| at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
| at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
| at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
| at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
| at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
|
| "pool-1-thread-2" prio=3 tid=0x01efd3f0 nid=0x274 waiting for monitor entry [0x4af7e000..0x4af7fa70]
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1299)
| - waiting to lock <0x794462d8> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
| at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
| - locked <0x7972c618> (a java.lang.Object)
| at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
| at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
| at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
| at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
| at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
| at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
| - locked <0x79730070> (a java.lang.Object)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
| at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
| at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
| at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
| at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
| at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
| at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
| at org.jboss.remoting.Client.invoke(Client.java:1634)
| at org.jboss.remoting.Client.invoke(Client.java:548)
| at org.jboss.remoting.Client.invokeOneway(Client.java:598)
| at org.jboss.remoting.Client.invokeOneway(Client.java:786)
| at org.jboss.remoting.Client.invokeOneway(Client.java:776)
| at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
| at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
| at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:473)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
| at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
| at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
| at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
| at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
| at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
| at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
| at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
| at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
| at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)
| at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
| at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
| at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
| at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
| at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
| at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
| at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
| at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
| at org.jboss.jms.client.JBossMessageProducer.publish(JBossMessageProducer.java:188)
| at com.datangmobile.oss.platform.jms.server.JmsSender.send(JmsSender.java:193)
| at com.datangmobile.oss.omc.server.cm.mmf.sliptree.SlipMitTree.bstRequestMO(SlipMitTree.java:337)
| at com.datangmobile.oss.omc.server.cm.mmf.sliptree.SlipMitTree.requestDynamicMO(SlipMitTree.java:513)
| at com.datangmobile.oss.omc.server.cm.mmf.sliptree.SlipNeTree.requestDynamicMO(SlipNeTree.java:207)
| at com.datangmobile.oss.omc.server.cm.ejb.SvrConfigMDB.neDynamicChange(SvrConfigMDB.java:255)
| at com.datangmobile.oss.omc.server.cm.ejb.SvrConfigMDB.processMessageWithThread(SvrConfigMDB.java:163)
| at com.datangmobile.oss.omc.server.cm.ejb.SvrConfigMDB$1.processMessage(SvrConfigMDB.java:147)
| at com.datangmobile.oss.platform.jms.msg.dispatch.AbstractProcessor.processMessages(AbstractProcessor.java:24)
| at com.datangmobile.oss.platform.jms.msg.dispatch.ProcessMsgWorker.run(MsgDispatcher.java:187)
| at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
| at java.lang.Thread.run(Thread.java:595)
|
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203),The thread got MessagingPostOffice's readwrite lock to route msg,and it locked MessagingQueue instance to deliver msg.
Unfortunately,the thread is waiting on socketwrite() forever(due to no tcp receive windows)
And one thread which holder MessagingQueue - locked <0x79730070> (a java.lang.Object) is waiting to lock <0x794462d8> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint) which holded by previous thread.
********************************************************************************************************************************
The delivey(post msg to subscriber or recevie msg form publisher) threads in JBM server are waiting to MessagingPostOffice's readwrite lock.
The threads can not process any msg due to waiting to lock.So the tcp receive window is full,and deadlock happens.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4191616#4191616
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4191616
More information about the jboss-user
mailing list