[
https://jira.jboss.org/jira/browse/JBMESSAGING-1699?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1699:
----------------------------------------------
We then added the following to jbm-configuration.xml:
<address-settings>
<address-setting match="#">
<max-size-bytes>10485760</max-size-bytes>
<page-size-bytes>1048576</page-size-bytes>
<drop-messages-when-full>true</drop-messages-when-full>
</address-setting>
</address-settings>
The idea here was to detect if the server crashing was being caused by it running out of
memory. The above lines should result in some log messages reporting low memory. After
restarting the server it crashed again in under half an hour (quite soon after I restarted
one of the clients for another reason). Unfortunately the logs were on FINE level and
there was too much output to trawl through to find the messages. So I changed the log
level back to INFO and restarted the server. It has then run fine for the last 13 hours
without crashing. I saw the following messages after startup for a few minutes and since
then there has been nothing in the logs and everything has run smoothly:
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,078 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
bee
n detected: Did not receive ping from /10.105.1.30:11404. It is likely the client has
exited or crashed without closing its connection, or the network between the s
erver and client has failed. The connection will now be closed. [code=3]
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,079 WARNING
[org.jboss.messaging.core.server.impl.ServerSessionImpl] Client connection failed,
clearin
g up resources for session e28f63f8-882c-11de-b548-00304863ffb4
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,079 WARNING
[org.jboss.messaging.core.server.impl.ServerSessionImpl] Cleared up resources for
session
e28f63f8-882c-11de-b548-00304863ffb4
[Thread-0 (group:JBM-scheduled-threads-2026561073)] 18:00:45,080 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
bee
n detected: Did not receive ping from /10.101.1.58:23493. It is likely the client has
exited or crashed without closing its connection, or the network between the s
erver and client has failed. The connection will now be closed. [code=3]
[Thread-3 (group:JBM-scheduled-threads-2026561073)] 18:00:47,617 SEVERE
[org.jboss.messaging.core.management.impl.ManagementServiceImpl]
Failed to call listener
java.lang.IllegalStateException: Cannot find queue info for queue
0beabc56-c181-4f2a-ba64-13624c0dcaafe19d3b25-882c-11de-a136-00e081341e12
at
org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.onNotification(PostOfficeImpl.java:387)
at
org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:647)
at
org.jboss.messaging.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:272)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:339)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleClose(ServerSessionImpl.java:2217)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.handleClose(ServerSessionImpl.java:882)
at
org.jboss.messaging.core.server.impl.ServerSessionImpl.connectionFailed(ServerSessionImpl.java:1177)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:393)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$FailedConnectionAction.run(RemotingServiceImpl.java:480)
at org.jboss.messaging.core.remoting.impl.Pinger.run(Pinger.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Server fails with message "Did not receive initial ping for
connection"
-----------------------------------------------------------------------
Key: JBMESSAGING-1699
URL:
https://jira.jboss.org/jira/browse/JBMESSAGING-1699
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Jeff Mesnil
Fix For: 2.0.0.beta5
Attachments: jbm-jstack-fail.txt, messaging.log.0.initial_ping1.FINE,
messaging.log.fail, transaudiostreamer.log.initping
I just upgraded to 2.0.0.BETA3 to 2.0.0.BETA4. After doing the upgrade I restarted some
of the JBM clients. Some of these were deadlocked and had to be stopped using "kill
-9" - i.e. they did not cleanly shutdown their jbm-related resources. This went fine
for about 10 clients, but at some point during this process the server went absolutely
nuts, logging hundreds of messages like:
[Thread-3 (group:JBM-scheduled-threads-1321312491)] 16:31:36,436 WARNING
[org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has
been detected Did not receive ping from client. It is likely a client has exited or
crashed without closing its connection, or the network between the server and client has
failed. The connection will now be closed.:3
There were also a number of exceptions in the server log. After this I couldn't
CTRL-C to stop the server but had to kill -9 the jbm process as it had locked up entirely.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira