[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
Adrian Woodhead (JIRA)
jira-events at lists.jboss.org
Tue Aug 4 09:34:29 EDT 2009
[ https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12478835#action_12478835 ]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
As far as I can tell all our clients have now been moved over to b4 and this problem is still occurring every few hours or so, requiring a JBM server restart. The server log files don't seem to contain any new information that isn't in the log that I have already attached. I do have some client stack traces for you (this is from a client that I am totally sure is running b4).
The first sign of an error in the client logs is this:
{code}
04-Aug-2009 12:08:37 org.jboss.messaging.core.logging.Logger error
SEVERE: Failed to handle failover
MessagingException[errorCode=3 message=Timed out waiting for response when sending packet 32]
at org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at org.jboss.messaging.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:734)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.reattachSessions(ConnectionManagerImpl.java:761)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:641)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
04-Aug-2009 12:08:37 org.jboss.messaging.core.logging.Logger warn
WARNING: Connection failure has been detected Did not receive ping from server:3
{code}
this is followed very soon by
{code}
[2009-08-04 12:08:37] ERROR (fm.last.jms.BaseJMSObject:139) - Exception in JMS RECEIVER for /topic/connectionControl
javax.jms.JMSException: MessagingException[errorCode=3 message=Did not receive ping from server]
at org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:541)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Did not receive ping from server]
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction.run(ConnectionManagerImpl.java:1142)
at org.jboss.messaging.core.remoting.impl.Pinger.run(Pinger.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
... 3 more
{code}
when our OnException handler is called we try to disconnect and then reconnect at which point the following seems to occur in disconnect:
{code}
[2009-08-04 12:10:10] WARN (fm.last.jms.BaseJMSObject:110) - Error closing JMS connection
javax.jms.JMSException: Timed out waiting for response when sending packet 74
at org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:753)
at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:318)
at org.jboss.messaging.jms.client.JBossMessageConsumer.close(JBossMessageConsumer.java:153)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:275)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Timed out waiting for response when sending packet 74]
... 22 more
{code}
and this in connect:
{code}
[2009-08-04 13:11:08] ERROR (fm.last.jms.JMSReceiver:34) - Error connecting RECEIVER for /topic/connectionControl connected
javax.jms.JMSException: Timed out waiting for response when sending packet 30
at org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:326)
at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:962)
at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:706)
at org.jboss.messaging.jms.client.JBossConnection.createSessionInternal(JBossConnection.java:458)
at org.jboss.messaging.jms.client.JBossConnection.createSession(JBossConnection.java:160)
at fm.last.jms.BaseJMSObject.connect(BaseJMSObject.java:83)
at fm.last.jms.JMSReceiver.connect(JMSReceiver.java:27)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:120)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Timed out waiting for response when sending packet 30]
... 23 more
{code}
When I restart the server the onException handler in the client gets called again and then it manages to reconnect and everything resumes correctly.
> Server fails with message "Did not receive initial ping for connection"
> -----------------------------------------------------------------------
>
> Key: JBMESSAGING-1699
> URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
> Project: JBoss Messaging
> Issue Type: Bug
> Affects Versions: 2.0.0.beta4
> Reporter: Adrian Woodhead
> Assignee: Tim Fox
> Fix For: 2.0.0
>
> Attachments: messaging.log.fail
>
>
> I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some of the JBM clients. Some of these were deadlocked and had to be stopped using "kill -9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine for about 10 clients, but at some point during this process the server went absolutely nuts, logging hundreds of messages like:
> [Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has been detected Did not receive ping from client. It is likely a client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed.:3
> There were also a number of exceptions in the server log. After this I couldn't CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list