[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"

Fri Aug 14 04:00:27 EDT 2009

    [ https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12480757#action_12480757 ] 

Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------

We then added the following to jbm-configuration.xml:

<address-settings>
     <address-setting match="#">
       <max-size-bytes>10485760</max-size-bytes>
       <page-size-bytes>1048576</page-size-bytes>
       <drop-messages-when-full>true</drop-messages-when-full>
     </address-setting>
  </address-settings>

The idea here was to detect if the server crashing was being caused by it running out of memory. The above lines should result in some log messages reporting low memory. After restarting the server it crashed again in under half an hour (quite soon after I restarted one of the clients for another reason). Unfortunately the logs were on FINE level and there was too much output to trawl through to find the messages. So I changed the log level back to INFO and restarted the server. It has then run fine for the last 13 hours without crashing. I saw the following messages after startup for a few minutes and since then there has been nothing in the logs and everything has run smoothly:

[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,078 WARNING [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl]  Connection failure has bee
n detected: Did not receive ping from /10.105.1.30:11404. It is likely the client has exited or crashed without closing its connection, or the network between the s
erver and client has failed. The connection will now be closed. [code=3]
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,079 WARNING [org.jboss.messaging.core.server.impl.ServerSessionImpl]  Client connection failed, clearin
g up resources for session e28f63f8-882c-11de-b548-00304863ffb4
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,079 WARNING [org.jboss.messaging.core.server.impl.ServerSessionImpl]  Cleared up resources for session
e28f63f8-882c-11de-b548-00304863ffb4
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,080 WARNING [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl]  Connection failure has bee
n detected: Did not receive ping from /10.101.1.58:23493. It is likely the client has exited or crashed without closing its connection, or the network between the s
erver and client has failed. The connection will now be closed. [code=3]
[Thread-3 (group:JBM-scheduled-threads-2026561073)] 18:00:47,617 SEVERE [org.jboss.messaging.core.management.impl.ManagementServiceImpl]  
Failed to call listener
java.lang.IllegalStateException: Cannot find queue info for queue 0beabc56-c181-4f2a-ba64-13624c0dcaafe19d3b25-882c-11de-a136-00e081341e12
        at org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.onNotification(PostOfficeImpl.java:387)
        at org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:647)
        at org.jboss.messaging.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:272)
        at org.jboss.messaging.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:339)
        at org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleClose(ServerSessionImpl.java:2217)
        at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleClose(ServerSessionImpl.java:882)
        at org.jboss.messaging.core.server.impl.ServerSessionImpl.connectionFailed(ServerSessionImpl.java:1177)
        at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:393)
        at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
        at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$FailedConnectionAction.run(RemotingServiceImpl.java:480)
        at org.jboss.messaging.core.remoting.impl.Pinger.run(Pinger.java:116)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

> Server fails with message "Did not receive initial ping for connection"
> -----------------------------------------------------------------------
>
>                 Key: JBMESSAGING-1699
>                 URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
>             Project: JBoss Messaging
>          Issue Type: Bug
>    Affects Versions: 2.0.0.beta4
>            Reporter: Adrian Woodhead
>            Assignee: Jeff Mesnil
>             Fix For: 2.0.0.beta5
>
>         Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE, messaging.log.fail, transaudiostreamer.log.initping
>
>
> I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some of the JBM clients. Some of these were deadlocked and had to be stopped using "kill -9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine for about 10 clients, but at some point during this process the server went absolutely nuts, logging hundreds of messages like:
> [Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl]  Connection failure has been detected Did not receive ping from client. It is likely a client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed.:3
> There were also a number of exceptions in the server log.  After this I couldn't CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira