[ http://jira.jboss.com/jira/browse/JBMESSAGING-1110?page=comments#action_1... ] Ovidiu Feodorov commented on JBMESSAGING-1110: ---------------------------------------------- Also, messages seem to be vanished from the database. I started with 50,000 10K persistent text messages stored in the queue "ovidiu" on the node with the ID 39 (looked up a clustered but not load-balanced connection factory from that node and got confirmation by calling getServerID()). The queue was deployed as "clustered" State of the database after killing the client (which became unresponsive): mysql> SELECT COUNT(MESSAGE_ID) FROM JBM_MSG_REF; +-------------------+ | COUNT(MESSAGE_ID) | +-------------------+ | 49288 | +-------------------+ mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 35528; (this is MIN(MESSAGE_ID)) +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | 0 | 35528 | NULL | C | 39070061203453943 | NULL | 0 | 0 | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 84912; (this is MAX(MESSAGE_ID)) +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | 0 | 84912 | NULL | C | 39070075545073791 | NULL | 0 | 0 | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ mysql> SELECT * FROM JBM_POSTOFFICE; (irrelevant destination names edited for privacy reasons) +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | JMS post office | 38 | *** | queue.*** | NULL | 27 | Y | N | | JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N | | JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N | | JMS post office | 38 | *** | queue.*** | NULL | 29 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 30 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 31 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 32 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 36 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 34 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 35 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 33 | Y | N | | JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 23 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 25 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 26 | Y | N | | JMS post office | 38 | *** | queue.*** | NULL | 24 | Y | N | | JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N | | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N | | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N | | JMS post office | 39 | *** | queue.*** | NULL | 12 | Y | N | | JMS post office | 39 | *** | queue.*** | NULL | 15 | Y | N | | JMS post office | 39 | *** | queue.*** | NULL | 13 | Y | N | | JMS post office | 39 | *** | queue.*** | NULL | 3 | Y | N | | JMS post office | 39 | *** | queue.*** | NULL | 5 | Y | N | | JMS post office | 39 | *** | queue.*** | NULL | 4 | Y | N | | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ 26 rows in set (0.00 sec) Please note that the channel corresponding to the "ovidiu" queue on node 39 vanished, and it seems there's no "ovidiu" queue on 39. The JMX console on node 39 became unresponsive as well (couldn't check whether the queue is visible via the JMX interface) Any attempt to connect to node 39 (even just to create a Connection and then close it) was unsuccessful . This is the client-side log on a connection attempt: ofeodorov@yamato$ mess ping @Timer-1 19:45:51,445 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51137],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:46:31,458 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51142],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:46:56,467 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51143],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:47:31,479 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51146],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:48:21,497 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51147],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:48:46,506 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51148],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:49:26,521 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51150],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:50:36,548 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51154],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:51:31,567 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51159],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:52:21,586 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51160],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:53:06,603 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51162],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:53:46,619 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51164],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection @Timer-1 19:54:26,633 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@74b957ea: detected failure on control connection Thread[control: Socket[addr=/10.152.1.39,port=3309,localport=51177],5,main] (a4812d-p364ba-f7qyex29-1-f7qyfgeu-5: requesting new control connection -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBMESSAGING-1110?page=all ] Tim Fox updated JBMESSAGING-1110: --------------------------------- Fix Version/s: Unscheduled -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBMESSAGING-1110?page=comments#action_1... ] Tim Fox commented on JBMESSAGING-1110: -------------------------------------- This could be related to a jboss remoting issue where remoting incorrectly terminates a connection under load since it didn't receive a ping. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBMESSAGING-1110?page=all ] Tim Fox closed JBMESSAGING-1110. -------------------------------- Resolution: Done -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira