[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
As far as I can tell all our clients have now been moved over to b4 and this problem is
still occurring every few hours or so, requiring a JBM server restart. The server log
files don't seem to contain any new information that isn't in the log that I have
already attached. I do have some client stack traces for you (this is from a client that I
am totally sure is running b4).
The first sign of an error in the client logs is this:
{code}
04-Aug-2009 12:08:37 org.jboss.messaging.core.logging.Logger error
SEVERE: Failed to handle failover
MessagingException[errorCode=3 message=Timed out waiting for response when sending packet
32]
at
org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:734)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.reattachSessions(ConnectionManagerImpl.java:761)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:641)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
04-Aug-2009 12:08:37 org.jboss.messaging.core.logging.Logger warn
WARNING: Connection failure has been detected Did not receive ping from server:3
{code}
this is followed very soon by
{code}
[2009-08-04 12:08:37] ERROR (fm.last.jms.BaseJMSObject:139) - Exception in JMS RECEIVER
for /topic/connectionControl
javax.jms.JMSException: MessagingException[errorCode=3 message=Did not receive ping from
server]
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:541)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Did not receive ping from server]
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction.run(ConnectionManagerImpl.java:1142)
at org.jboss.messaging.core.remoting.impl.Pinger.run(Pinger.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
... 3 more
{code}
when our OnException handler is called we try to disconnect and then reconnect at which
point the following seems to occur in disconnect:
{code}
[2009-08-04 12:10:10] WARN (fm.last.jms.BaseJMSObject:110) - Error closing JMS connection
javax.jms.JMSException: Timed out waiting for response when sending packet 74
at
org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at
org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:753)
at
org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:318)
at
org.jboss.messaging.jms.client.JBossMessageConsumer.close(JBossMessageConsumer.java:153)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:275)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Timed out waiting for response when
sending packet 74]
... 22 more
{code}
and this in connect:
{code}
[2009-08-04 13:11:08] ERROR (fm.last.jms.JMSReceiver:34) - Error connecting RECEIVER for
/topic/connectionControl connected
javax.jms.JMSException: Timed out waiting for response when sending packet 30
at
org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:326)
at
org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:962)
at
org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:706)
at
org.jboss.messaging.jms.client.JBossConnection.createSessionInternal(JBossConnection.java:458)
at
org.jboss.messaging.jms.client.JBossConnection.createSession(JBossConnection.java:160)
at fm.last.jms.BaseJMSObject.connect(BaseJMSObject.java:83)
at fm.last.jms.JMSReceiver.connect(JMSReceiver.java:27)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:120)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Timed out waiting for response when
sending packet 30]
... 23 more
{code}
When I restart the server the onException handler in the client gets called again and then
it manages to reconnect and everything resumes correctly.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL:
https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira