[jboss-jira] [JBoss JIRA] (JBREM-1308) MicroSocketClientInvoker hangs on SocketInputStream.read causing deadlock in MessageSucker.setConsuming
Ron Sigal (JIRA)
jira-events at lists.jboss.org
Tue Jul 17 16:27:07 EDT 2012
[ https://issues.jboss.org/browse/JBREM-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706448#comment-12706448 ]
Ron Sigal commented on JBREM-1308:
----------------------------------
Hi Giovanni,
As far as I can see, this situation isn't a true deadlock. You have
* WorkManager(2)-42959 waiting to acquire <0x00000006012cdd50> (a java.lang.Object)
* where <0x00000006012cdd50> (a java.lang.Object) is held by Thread-5167
* where Thread-5167 is waiting to acquire <0x0000000605960788> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker)
* where <0x0000000605960788> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker) is held by Thread-5163
* where Thread-5163 is hanging in a socket read().
Unlike a true deadlock, there isn't a circular contention for locks. This situation is caused by a hanging socket read(), which presumably isn't timing out because the socket is configured with timeout set to 0. In fact, 0 is the default timeout for most versions of JBossMessaging. If you look in $JBOSS_HOME/server/$CONFIG/deploy/remoting-bisocket-service.xml, you should see a line like
<attribute name="timeout" isParam="true">0</attribute>
If you change 0 to a positive value (in microseconds), e.g., 30000, or whatever is suitable, I think the situation should go away.
Unless you have any information that is contrary to my analysis, I'm going to close this issue in the next day or so.
Thanks,
Ron
> MicroSocketClientInvoker hangs on SocketInputStream.read causing deadlock in MessageSucker.setConsuming
> -------------------------------------------------------------------------------------------------------
>
> Key: JBREM-1308
> URL: https://issues.jboss.org/browse/JBREM-1308
> Project: JBoss Remoting
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: transport
> Affects Versions: 2.2.4, 2.5.4.SP3
> Environment: jboss-4.2.3.GA with jboss-messaging-1.4.5.GA and latest jboss-remoting-2_2_4,
> 3 node cluster, jms clustered queues
> Reporter: Giovanni Pelosi
> Assignee: Ron Sigal
> Fix For: 2.2.4.SP1, 2.5.4.SP4
>
> Attachments: jstack-20120604-0501.log, jstack-20120604-0502.log, jstack-20120604-0503.log
>
>
> In production enviroment, our jboss line of 3 clustered servers randomly (once a week) seemes to hang, refusing to reply to remote jms clients.
> Taking stack trace of all jboss instances (jstack) we always see the same pattern of deadlock:
> A non-pooled thread, started internally by remoting, is trying to recover a "connectionExeption" condition
> (ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120)),
> but the task hangs (without timeout) on a socket read:
> (MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077))
> This non-terminating network operation holds a lock, acquired in
> MessageSucker.suspend(MessageSucker.java:225)
> that in turns cause deadlock in messaging layer:
> org.jboss.messaging.core.impl.MessagingQueue.informSuckers(MessagingQueue.java:566)
> causing blocks in all jms traffic.
> Offending thread stack:
> -------------------------------------------------
> "Thread-5163" daemon prio=10 tid=0x00007f1bc4d10800 nid=0x2b70 runnable [0x00007f1af61e0000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:146)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> - locked <0x00000006d23445f8> (a java.io.BufferedInputStream)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077)
> at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682)
> at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
> at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
> at org.jboss.remoting.Client.invoke(Client.java:1925)
> at org.jboss.remoting.Client.invoke(Client.java:786)
> at org.jboss.remoting.Client.invoke(Client.java:774)
> at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
> at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
> at org.jboss.jms.client.delegate.ClientConsumerDelegate.org$jboss$jms$client$delegate$ClientConsumerDelegate$close$aop(ClientConsumerDelegate.java:122)
> at org.jboss.jms.client.delegate.ClientConsumerDelegate$close_N4742752445160157748.invokeNext(ClientConsumerDelegate$close_N4742752445160157748.java)
> at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
> at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
> at org.jboss.jms.client.delegate.ClientConsumerDelegate$close_N4742752445160157748.invokeNext(ClientConsumerDelegate$close_N4742752445160157748.java)
> at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
> at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
> at org.jboss.jms.client.delegate.ClientConsumerDelegate$close_N4742752445160157748.invokeNext(ClientConsumerDelegate$close_N4742752445160157748.java)
> at org.jboss.jms.client.delegate.ClientConsumerDelegate.close(ClientConsumerDelegate.java)
> at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.suspend(MessageSucker.java:225)
> - locked <0x0000000605960788> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker)
> at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.cleanupConnection(ClusterConnectionManager.java:804)
> - locked <0x0000000605960ca8> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo)
> at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.onException(ClusterConnectionManager.java:785)
> at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120)
> at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498)
> -------------------------------------------------
> JMS Layer main lock
> -------------------------------------------------
> "Thread-5167" daemon prio=10 tid=0x00007f1be0069800 nid=0x2b82 waiting for monitor entry [0x00007f1afac2b000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.setConsuming(MessageSucker.java:288)
> - waiting to lock <0x0000000605960788> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker)
> at org.jboss.messaging.core.impl.MessagingQueue.informSuckers(MessagingQueue.java:566)
> at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:516)
> at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:375)
> - locked <0x00000006012cdd50> (a java.lang.Object)
> at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1574)
> at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:120)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> -------------------------------------------------
> Sample of deadlocked JMS Client Thread:
> -------------------------------------------
> "WorkManager(2)-42959" daemon prio=10 tid=0x00007f1bb806c000 nid=0x2b6c waiting for monitor entry [0x00007f1adf230000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:242)
> - waiting to lock <0x00000006012cdd50> (a java.lang.Object)
> at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2345)
> at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:515)
> at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:769)
> at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
> at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
> at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
> at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
> at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
> at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
> at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
> at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
> at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
> at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:862)
> at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1901)
> at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
> at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1912)
> at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:832)
> at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
> at org.jboss.remoting.Client.invoke(Client.java:1925)
> at org.jboss.remoting.Client.invoke(Client.java:786)
> at org.jboss.remoting.Client.invokeOneway(Client.java:837)
> at org.jboss.remoting.Client.invokeOneway(Client.java:1023)
> at org.jboss.remoting.Client.invokeOneway(Client.java:1014)
> at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
> at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
> at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495)
> at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
> at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
> at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
> at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
> at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
> at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
> at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
> at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
> at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
> at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
> at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
> at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276)
> at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
> at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
> at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
> at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
> at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
> at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
> at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165)
> at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208)
> at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146)
> at com.pirelli.pmp.jms.JmsClient.send(JmsClient.java:457)
> at com.pirelli.pmp.jms.JmsClient.send(JmsClient.java:606)
> at com.pirelli.pmp.jms.JmsClient.send(JmsClient.java:655)
> at com.pirelli.pmp.jms.GuiResponseQueueHelper.sendReplyMessage(GuiResponseQueueHelper.java:31)
> at com.pirelli.pmp.jms.GuiResponseQueueHelper.sendReplyMessage(GuiResponseQueueHelper.java:26)
> at com.pirelli.pmp.mdb.ActivityStarterMdb.replyToGUI(ActivityStarterMdb.java:153)
> at com.pirelli.pmp.mdb.ActivityStarterMdb.onMessage(ActivityStarterMdb.java:95)
> at sun.reflect.GeneratedMethodAccessor1335.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
> at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
> at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
> at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
> at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
> at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
> at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
> at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
> at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
> at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
> at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
> at $Proxy117.onMessage(Unknown Source)
> at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
> at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
> at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
> at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
> at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
> at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
> at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
> at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
> at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
> at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
> at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
> at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
> at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
> at java.lang.Thread.run(Thread.java:679)
> -------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list