[JBoss JIRA] Created: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
11:53 a.m.
Server fails with message "Did not receive initial ping for connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some of
the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up
entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:55 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead updated JBMESSAGING-1699:
-----------------------------------------
Attachment: messaging.log.fail
Log file from start up until the server went insane.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Attachments: messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:05 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Clebert Suconic commented on JBMESSAGING-1699:
----------------------------------------------
Some relevant conversation we had on IRC:
11:57:51 AM) adrianW: clebert - this is what I did:
(11:58:00 AM) adrianW: 1. upgrade all clients to b4 (still talking to b3 server)
(11:58:11 AM) adrianW: 2. stop b3 server
(11:58:22 AM) adrianW: 3. start b4 server with same config, port
(11:58:34 AM) adrianW: 4. clients reconnect and everything now on b4
(11:58:41 AM) adrianW: 5. restart b4 server just to be sure
(11:58:55 AM) adrianW: 6. some clients did not seem to reconnect, so I restarted them
(11:59:10 AM) adrianW: at some point during 6 the crash happened
(12:00:57 PM) clebert: adrianW: ok.. I see...
(12:00:57 PM) clebert: One thing though.. until we are final (GA).. I don't think we
are guaranteeing version compatibility between the wire protocol.
(12:00:57 PM) clebert: I don' t think anything changed between Beta3 and 4.. but just
to let you know
(12:01:12 PM) adrianW: yeah, I heard that
(12:01:21 PM) adrianW: I tried it out on a test client/server first and it seemed ok
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Attachments: messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:07 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1699:
--------------------------------------
Adrian-
Can you attach a test program that replicates the issue?
I also notice Netty exception in the logs - I will ask Trustin what these mean.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Attachments: messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:40 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1699:
--------------------------------------
Actually the IRC snippet is misleading.
Actually a lot changed between b3 and b4 in the area of pinging and netty versions.
I suspect the problem has occurred because of different versions of clients and servers
being used together.
Also, it seems that the JBM version number was not changed between b3 and b4 releases
making it harder to track down errors like this.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Attachments: messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:10 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Tim Fox updated JBMESSAGING-1699:
---------------------------------
Fix Version/s: 2.0.0
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:34 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
As far as I can tell all our clients have now been moved over to b4 and this problem is
still occurring every few hours or so, requiring a JBM server restart. The server log
files don't seem to contain any new information that isn't in the log that I have
already attached. I do have some client stack traces for you (this is from a client that I
am totally sure is running b4).
The first sign of an error in the client logs is this:
{code}
04-Aug-2009 12:08:37 org.jboss.messaging.core.logging.Logger error
SEVERE: Failed to handle failover
MessagingException[errorCode=3 message=Timed out waiting for response when sending packet
32]
at
org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:734)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.reattachSessions(ConnectionManagerImpl.java:761)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:641)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
04-Aug-2009 12:08:37 org.jboss.messaging.core.logging.Logger warn
WARNING: Connection failure has been detected Did not receive ping from server:3
{code}
this is followed very soon by
{code}
[2009-08-04 12:08:37] ERROR (fm.last.jms.BaseJMSObject:139) - Exception in JMS RECEIVER
for /topic/connectionControl
javax.jms.JMSException: MessagingException[errorCode=3 message=Did not receive ping from
server]
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:541)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Did not receive ping from server]
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction.run(ConnectionManagerImpl.java:1142)
at org.jboss.messaging.core.remoting.impl.Pinger.run(Pinger.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
... 3 more
{code}
when our OnException handler is called we try to disconnect and then reconnect at which
point the following seems to occur in disconnect:
{code}
[2009-08-04 12:10:10] WARN (fm.last.jms.BaseJMSObject:110) - Error closing JMS connection
javax.jms.JMSException: Timed out waiting for response when sending packet 74
at
org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at
org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:753)
at
org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:318)
at
org.jboss.messaging.jms.client.JBossMessageConsumer.close(JBossMessageConsumer.java:153)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:275)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Timed out waiting for response when
sending packet 74]
... 22 more
{code}
and this in connect:
{code}
[2009-08-04 13:11:08] ERROR (fm.last.jms.JMSReceiver:34) - Error connecting RECEIVER for
/topic/connectionControl connected
javax.jms.JMSException: Timed out waiting for response when sending packet 30
at
org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:326)
at
org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:962)
at
org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:706)
at
org.jboss.messaging.jms.client.JBossConnection.createSessionInternal(JBossConnection.java:458)
at
org.jboss.messaging.jms.client.JBossConnection.createSession(JBossConnection.java:160)
at fm.last.jms.BaseJMSObject.connect(BaseJMSObject.java:83)
at fm.last.jms.JMSReceiver.connect(JMSReceiver.java:27)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:120)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: MessagingException[errorCode=3 message=Timed out waiting for response when
sending packet 30]
... 23 more
{code}
When I restart the server the onException handler in the client gets called again and then
it manages to reconnect and everything resumes correctly.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:36 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead updated JBMESSAGING-1699:
-----------------------------------------
Attachment: jbm-jstack-fail.txt
FWIW I have attached a jstack of the server process when it goes into this fail mode and
stops responding to CTRL-C.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: jbm-jstack-fail.txt, messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:55 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1699:
--------------------------------------
According to the log the client has failed then tried to fail over onto the backup server.
Can you explain a little about your topology? I.e. how many servers, how many backup
servers etc?
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: jbm-jstack-fail.txt, messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:34 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
Sure, currently we have *one* JBM server and around 20 clients. Each client is both a
producer and a receiver on a single topic.
The jbm-jxm.xml contains the following elements related to failover:
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>1.0</retry-interval-multiplier>
<reconnect-attempts>-1</reconnect-attempts>
<failover-on-server-shutdown>true</failover-on-server-shutdown>
<call-timeout>5000</call-timeout>
So from what I understood if this server goes down clients will keep trying to reconnect
to it until it comes back (i.e. its not so much failover as
"keep-trying-to-reconnect")
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: jbm-jstack-fail.txt, messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:18 a.m.
New subject: [JBoss JIRA] Assigned: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Jeff Mesnil reassigned JBMESSAGING-1699:
----------------------------------------
Assignee: Jeff Mesnil (was: Tim Fox)
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0
Attachments: jbm-jstack-fail.txt, messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:50 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead updated JBMESSAGING-1699:
-----------------------------------------
Attachment: messaging.log.0.initial_ping1.FINE
JBM server log on FINE level from fresh startup to crash.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:54 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead updated JBMESSAGING-1699:
-----------------------------------------
Attachment: transaudiostreamer.log.initping
Client log for around the same time as the server crashed.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:07 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
The server crashed again last night and was left in this state for a few hours before I
got to restart it. I noticed two new exceptions repeated in the log:
[New I/O server worker #1-2] 17:53:39,823 SEVERE
[org.jboss.messaging.core.server.impl.ServerSessionImpl] Failed to acknowledge
java.lang.NullPointerException
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleExpired(ServerSessionImpl.java:1623)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.handleExpired(ServerSessionImpl.java:478)
at
org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:152)
at org.jboss.messaging.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:621)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:364)
at
org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:483)
at
org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:75)
at
org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:324)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:306)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:223)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:87)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:330)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:203)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:113)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
at
org.jboss.messaging.integration.transports.netty.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:184)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
and
[New I/O server worker #1-1] 17:56:29,619 SEVERE
[org.jboss.messaging.core.server.impl.ServerSessionPacketHandler] Caught unexpected
exception
java.lang.NullPointerException
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.handleCloseConsumer(ServerSessionImpl.java:921)
at
org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:257)
at org.jboss.messaging.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:621)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:364)
at
org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:483)
at
org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:75)
at
org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:324)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:306)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:223)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:87)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:330)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:203)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:113)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
at
org.jboss.messaging.integration.transports.netty.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:184)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:18 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Tim Fox updated JBMESSAGING-1699:
---------------------------------
JBoss Forum Reference:
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4249444#...
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:20 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1699:
--------------------------------------
As per forum thread
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4249459#...
We replicated the issue and it seems to occur because the server is running out of RAM.
Messages are being received at the server at a much higher rate than they can be
distributed out to all the consumers.
Since no paging params have been set for the topic address, the server will eventually run
out of RAM and crash and burn.
To prevent this I set paging params on the topic address and successfully ran 40 million
messages through without problem on my laptop.
Jeff has done the same on the messaging lab.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:52 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
Upgraded to JBM trunk from 2009-08-12. An hour or so after upgrading all the clients, the
server crashed again, here are some fresh logs:
[Thread-3 (group:JBM-scheduled-threads-999186587)] 16:23:42,871 WARNING
[org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl] Did not receive
initial
ping from /10.105.1.32:50019, connection will be closed
[Thread-3 (group:JBM-scheduled-threads-999186587)] 16:23:42,872 WARNING
[org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl] Did not receive
initial
ping from /10.105.1.29:47834, connection will be closed
[New I/O server worker #1-2] 16:23:42,872 WARNING
[org.jboss.netty.channel.socket.nio.NioWorker] Unexpected exception in the selector
loop.
org.jboss.netty.channel.ChannelException: Failed to register a socket to the selector.
at
org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:754)
at
org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:260)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:199)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
at
org.jboss.messaging.integration.transports.netty.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:184)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedChannelException
at
java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:252)
at
org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:739)
... 7 more
[Thread-4 (group:JBM-scheduled-threads-999186587)] 16:23:42,891 SEVERE
[org.jboss.messaging.core.management.impl.ManagementServiceImpl] Failed to call listener
java.lang.IllegalStateException: Cannot find queue info for queue
e737ed9e-52fe-4160-a651-23b53978db2808429257-87e3-11de-8e7d-00e081341e12
at
org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.onNotification(PostOfficeImpl.java:387)
at
org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:647)
at
org.jboss.messaging.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:272)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:339)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleClose(ServerSessionImpl.java:2217)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.handleClose(ServerSessionImpl.java:882)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.connectionFailed(ServerSessionImpl.java:1177)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:393)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$FailedConnectionAction.run(RemotingServiceImpl.java:480)
at org.jboss.messaging.core.remoting.impl.Pinger.run(Pinger.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
[New I/O server worker #1-2] 16:23:05,454 WARNING
[org.jboss.netty.channel.socket.nio.NioWorker] Unexpected exception in the selector
loop.
org.jboss.netty.channel.ChannelException: Failed to register a socket to the selector.
at
org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:754)
at
org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:260)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:199)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
at
org.jboss.messaging.integration.transports.netty.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:184)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedChannelException
at
java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:252)
at
org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:739)
... 7 more
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:55 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
In the client logs at the same time we saw the following:
2009-08-13 16:22:33] ERROR (fm.last.jms.JMSReceiver:34) - Error connecting RECEIVER for
/topic/connectionControl connected
javax.jms.JMSException: Timed out waiting for response when sending packet 30
at
org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:312)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:326)
at
org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:977)
at
org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:721)
at
org.jboss.messaging.jms.client.JBossConnection.authorize(JBossConnection.java:519)
at
org.jboss.messaging.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:535)
at
org.jboss.messaging.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:115)
at
org.jboss.messaging.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:110)
at fm.last.jms.BaseJMSObject.connect(BaseJMSObject.java:82)
at fm.last.jms.JMSReceiver.connect(JMSReceiver.java:27)
at fm.last.jms.ConnectionMonitor.run(ConnectionMonitor.java:48)
Caused by: MessagingException[errorCode=3 message=Timed out waiting for response when
sending packet 30]
... 11 more
[2009-08-13 16:22:33] WARN (fm.last.jms.ConnectionMonitor:53) - Connection down, trying
again in 10000ms...
[2009-08-13 16:22:36] WARN (fm.last.jms.QueueingProducer:86) - JMS connection down,
dropping 10 messages (/topic/connectionControl)
13-Aug-2009 16:22:39 org.jboss.messaging.core.logging.Logger warn
WARNING: I'm closing a connection you left open. Please make sure you close all
connections explicitly before letting them go out of scope!
13-Aug-2009 16:22:39 org.jboss.messaging.core.logging.Logger warn
Interesting that for some reason it looks like the client connections weren't closed -
when onException is called the first thing I do is a disconnect which closes the
connection.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:58 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1699:
--------------------------------------
Adrian- is your latest failure with or without the paging parameters we discussed
yesterday?
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
We then added the following to jbm-configuration.xml:
<address-settings>
<address-setting match="#">
<max-size-bytes>10485760</max-size-bytes>
<page-size-bytes>1048576</page-size-bytes>
<drop-messages-when-full>true</drop-messages-when-full>
</address-setting>
</address-settings>
The idea here was to detect if the server crashing was being caused by it running out of
memory. The above lines should result in some log messages reporting low memory. After
restarting the server it crashed again in under half an hour (quite soon after I restarted
one of the clients for another reason). Unfortunately the logs were on FINE level and
there was too much output to trawl through to find the messages. So I changed the log
level back to INFO and restarted the server. It has then run fine for the last 13 hours
without crashing. I saw the following messages after startup for a few minutes and since
then there has been nothing in the logs and everything has run smoothly:
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,078 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
bee
n detected: Did not receive ping from /10.105.1.30:11404. It is likely the client has
exited or crashed without closing its connection, or the network between the s
erver and client has failed. The connection will now be closed. [code=3]
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,079 WARNING
[org.jboss.messaging.core.server.impl.ServerSessionImpl] Client connection failed,
clearin
g up resources for session e28f63f8-882c-11de-b548-00304863ffb4
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,079 WARNING
[org.jboss.messaging.core.server.impl.ServerSessionImpl] Cleared up resources for
session
e28f63f8-882c-11de-b548-00304863ffb4
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,080 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
bee
n detected: Did not receive ping from /10.101.1.58:23493. It is likely the client has
exited or crashed without closing its connection, or the network between the s
erver and client has failed. The connection will now be closed. [code=3]
[Thread-3 (group:JBM-scheduled-threads-2026561073)] 18:00:47,617 SEVERE
[org.jboss.messaging.core.management.impl.ManagementServiceImpl]
Failed to call listener
java.lang.IllegalStateException: Cannot find queue info for queue
0beabc56-c181-4f2a-ba64-13624c0dcaafe19d3b25-882c-11de-a136-00e081341e12
at
org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.onNotification(PostOfficeImpl.java:387)
at
org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:647)
at
org.jboss.messaging.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:272)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:339)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleClose(ServerSessionImpl.java:2217)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.handleClose(ServerSessionImpl.java:882)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.connectionFailed(ServerSessionImpl.java:1177)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:393)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$FailedConnectionAction.run(RemotingServiceImpl.java:480)
at org.jboss.messaging.core.remoting.impl.Pinger.run(Pinger.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:35 a.m.
New subject: [JBoss JIRA] Closed: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Tim Fox closed JBMESSAGING-1699.
--------------------------------
Resolution: Done
We believe this issue should be fixed now.
Adrian - can you try trunk out in your setup?
You can monitor memory by using the new parameter that jeff added - this will tell you if
you are running out of RAM when the problem occurs
If the problem re-occurs we will reopen the task.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:37 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1699) Server fails with message "Did not receive initial ping for connection"
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
I doubt I will have time to try the trunk this week. Interestingly, the trunk build from
last week has been stable since the second restart of the server, the server hasn't
gone down once. I am keeping my eye on it and will report if anything new comes up.
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6185
days inactive
6201
days old
21 comments
4 participants
participants (4)
-
Adrian Woodhead (JIRA) -
Clebert Suconic (JIRA)
-
Jeff Mesnil (JIRA)
-
Tim Fox (JIRA)