[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"

[Adrian Woodhead (JIRA)]

    [ https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12480751#action_12480751 ] 

Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------

Upgraded to JBM trunk from 2009-08-12. An hour or so after upgrading all the clients, the server crashed again, here are some fresh logs:

[Thread-3 (group:JBM-scheduled-threads-999186587)] 16:23:42,871 WARNING [org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl]  Did not receive initial
 ping from /10.105.1.32:50019, connection will be closed
[Thread-3 (group:JBM-scheduled-threads-999186587)] 16:23:42,872 WARNING [org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl]  Did not receive initial
 ping from /10.105.1.29:47834, connection will be closed
[New I/O server worker #1-2] 16:23:42,872 WARNING [org.jboss.netty.channel.socket.nio.NioWorker]  Unexpected exception in the selector loop.
org.jboss.netty.channel.ChannelException: Failed to register a socket to the selector.
        at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:754)
        at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:260)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:199)
        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
        at org.jboss.messaging.integration.transports.netty.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:184)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedChannelException
        at java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:252)
        at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:739)
        ... 7 more
[Thread-4 (group:JBM-scheduled-threads-999186587)] 16:23:42,891 SEVERE [org.jboss.messaging.core.management.impl.ManagementServiceImpl]  Failed to call listener
java.lang.IllegalStateException: Cannot find queue info for queue e737ed9e-52fe-4160-a651-23b53978db2808429257-87e3-11de-8e7d-00e081341e12
        at org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.onNotification(PostOfficeImpl.java:387)
        at org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:647)
        at org.jboss.messaging.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:272)
        at org.jboss.messaging.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:339)
        at org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleClose(ServerSessionImpl.java:2217)
        at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleClose(ServerSessionImpl.java:882)
        at org.jboss.messaging.core.server.impl.ServerSessionImpl.connectionFailed(ServerSessionImpl.java:1177)
        at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:393)
        at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
        at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$FailedConnectionAction.run(RemotingServiceImpl.java:480)
        at org.jboss.messaging.core.remoting.impl.Pinger.run(Pinger.java:116)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)


[New I/O server worker #1-2] 16:23:05,454 WARNING [org.jboss.netty.channel.socket.nio.NioWorker]  Unexpected exception in the selector loop.
org.jboss.netty.channel.ChannelException: Failed to register a socket to the selector.
        at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:754)
        at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:260)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:199)
        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
        at org.jboss.messaging.integration.transports.netty.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:184)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedChannelException
        at java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:252)
        at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:739)
        ... 7 more



> Server fails with message "Did not receive initial ping for connection"
> -----------------------------------------------------------------------
>
>                 Key: JBMESSAGING-1699
>                 URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
>             Project: JBoss Messaging
>          Issue Type: Bug
>    Affects Versions: 2.0.0.beta4
>            Reporter: Adrian Woodhead
>            Assignee: Jeff Mesnil
>             Fix For: 2.0.0.beta5
>
>         Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE, messaging.log.fail, transaudiostreamer.log.initping
>
>
> I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some of the JBM clients. Some of these were deadlocked and had to be stopped using "kill -9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine for about 10 clients, but at some point during this process the server went absolutely nuts, logging hundreds of messages like:
> [Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl]  Connection failure has been detected Did not receive ping from client. It is likely a client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed.:3
> There were also a number of exceptions in the server log.  After this I couldn't CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira