[jboss-user] [JBoss Messaging] - Re: BisocketClientInvoker deadlock

azafran do-not-reply at jboss.com
Wed May 14 15:27:57 EDT 2008


I'll elaborate on the previous:

The "Notify Thread" locks other threads since it acquires the lock on UserConnectionImpl. What we're worried about is why this thread "hangs" while trying to send a message. When reproducing this phenomena, we've witnessed two scenarios:
1) "Notify Thread" hangs on MessagingPostOffice line 2075, as previously posted
2) "Notify Thread" hangs on BisocketClientInvoker line 473, as follows

  | "Notify Thread" prio=10 tid=0x029796e8 nid=0x58 in Object.wait() [0xda78c000..0xda78fb70]
  | 	at java.lang.Object.wait(Native Method)
  | 	- waiting on <0x4df481b8> (a java.util.HashSet)
  | 	at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)
  | 	- locked <0x4df481b8> (a java.util.HashSet)
  | 	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)
  | 	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)
  | 	at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
  | 	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
  | 	at org.jboss.remoting.Client.invoke(Client.java:1634)
  | 	at org.jboss.remoting.Client.invoke(Client.java:548)
  | 	at org.jboss.remoting.Client.invokeOneway(Client.java:598)
  | 	at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
  | 	at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
  | 	at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
  | 	at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
  | 	- locked <0x4e4b8988> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
  | 	at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
  | 	- locked <0x4e4c1d08> (a java.lang.Object)
  | 	at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
  | 	at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
  | 	at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
  | 	at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
  | 	at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
  | 	at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
  | 	- locked <0x4e4ba7f0> (a java.lang.Object)
  | 	at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
  | 	at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
  | 	at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
  | 	at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
  | 	at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
  | 	at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
  | 	at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
  | 	at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
  | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  | 	at java.lang.reflect.Method.invoke(Method.java:585)
  | 	at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
  | 	at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
  | 	at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
  | 	at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
  | 	at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
  | 	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
  | 	at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
  | 	at org.jboss.remoting.Client.invoke(Client.java:1634)
  | 	at org.jboss.remoting.Client.invoke(Client.java:548)
  | 	at org.jboss.remoting.Client.invoke(Client.java:536)
  | 	at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
  | 	at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
  | 	at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
  | 	at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
  | 	at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
  | 	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
  | 	at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
  | 	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
  | 	at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
  | 	at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
  | 	at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
  | 	at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
  | 	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
  | 	at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
  | 	at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
  | 	at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
  | 	at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
  | 	at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
  | 	at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
  | 	at com.cisco.nm.cmp.gjc.cm.UserConnectionImpl.sendMsg(UserConnectionImpl.java:319)
  | 	- locked <0x47864780> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
  | 	at com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.send(GWCommandListenerImpl.java:207)
  | 	- locked <0x4e2e5838> (a com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl)
  | 	at com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.notifications(GWCommandListenerImpl.java:179)
  | 	- locked <0x4e2e5838> (a com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl)
  | 	at com.cisco.nm.cmp.client.j2ee.connector.DnaCommand$CommandListener.notify(DnaCommand.java:838)
  | 	at com.cisco.nm.cmp.gw.ptpclient.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:49)
  | 	at com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.notifyListeners(NotifyThread.java:193)
  | 	at com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.run(NotifyThread.java:148)
  | 

There are many JMS clients connecting to the server and ungraceful disconnections are common, due to poor network, firewall etc.. It seems that the BisocketClientInvoker is waiting on the sockets set forever.


View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4150865#4150865

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4150865



More information about the jboss-user mailing list