6:28 a.m.
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1743?page=com.atlassian.ji...
]
Howard Gao commented on JBMESSAGING-1743:
-----------------------------------------
Hi Zachary,
I'm trying to analyse the trace and I found how it hang, the trace showed the
following sequence of events:
Message producer is sending out a message. JBM has two interceptors in the way,
ClosedInterceptor and FailoverValveInterceptor. In the calling stack the ClosedInterceptor
is ahead of FailoverValveInterceptor.
The ClosedInterceptor makes sure any invocation is completed before closing of the object
on which the invocation is made. It does this by maintaining a counter inUseCount. It is
incremented by one before each invocation (like send()) and dedremented by one when the
invocation is finished. If there is some on-going (not finished) invocation when the
object is being closed, the interceptor will wait on the inUserCounter until it reaches
zero.
The FailoverValveInterceptor is used to coordinate failover process and catch connection
failures. If there is a failover going on while an invocation is being made, the
interceptor will hold the invocation until failover completes. If there is a connection
failure during the invocation, it will notify the failover handler to fail over and then
redo the invocation.
In this case, the send() call first passed the ClosedInterceptor normally (the inUseCount
> 0) and went into FailoverValveInterceptor, where the connection failure happened. So
it immediately initiate the failover handling. In the failover handling, first it tried to
close the failed connection, which resulted in closing its session first. And the closing
of session will result in closing of its producer first.
When the producer's close() is called, its ClosedInterceptor intercepted the call and
found there is an ongoing invocation (inUseCount > 0) and so it waited for the
inUseCount to be zero. But this is never going to happen as this thread was the calling
thread of send(). If it waits there there is no way for the inUseCount go back to zero.
I'm thinking how to reproduce this and if my analysis is correct, how to come up a
proper fix.
Thanks
Howard
JBoss Messaging halting on queueSender.send
-------------------------------------------
Key: JBMESSAGING-1743
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1743
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP08
Environment: Virtualized RHEL 5 on IBM blades
Reporter: Zachary Bonig
Assignee: Howard Gao
Fix For: 1.4.0.SP3.CP10
Intermittently, messages sent through queueSender.send will fail and hang indefinitely
(until messaging server restart). These messages always hang in the same place:
Thread: http-webllm31%2F10.10.22.25-8080-1 : priority:5, demon:true, threadId:187,
threadState:WAITING, lockName:org.jboss.jms.client.container.ClosedInterceptor@15f2c
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.jms.client.container.ClosedInterceptor.checkCloseAlreadyDone(ClosedInterceptor.java:245)
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:142)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientProducerDelegate$close_N4742752445160157748.invokeNext(ClientProducerDelegate$close_N4742752445160157748.java)
org.jboss.jms.client.delegate.ClientProducerDelegate.close(ClientProducerDelegate.java)
org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:306)
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:165)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientSessionDelegate$closing_2473194355759371067.invokeNext(ClientSessionDelegate$closing_2473194355759371067.java)
org.jboss.jms.client.delegate.ClientSessionDelegate.closing(ClientSessionDelegate.java)
org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:305)
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:165)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientConnectionDelegate$closing_2473194355759371067.invokeNext(ClientConnectionDelegate$closing_2473194355759371067.java)
org.jboss.jms.client.delegate.ClientConnectionDelegate.closing(ClientConnectionDelegate.java)
org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:208)
org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:124)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276)
org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)
org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165)
org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208)
org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146)
com.tc.csclient.handler.SyncCSCommunicationHandler.getReplyThroughStaticQueue(SyncCSCommunicationHandler.java:249)
............ remainder of stack trace removed for Clarity................
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira