[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1112) Consumption with highly concurrent clients lock a JBM server node.

Consumption with highly concurrent clients lock a JBM server node. ------------------------------------------------------------------ Key: JBMESSAGING-1112 URL: http://jira.jboss.com/jira/browse/JBMESSAGING-1112 Project: JBoss Messaging Issue Type: Bug Affects Versions: 1.4.0.GA Reporter: Ovidiu Feodorov Assigned To: Tim Fox Priority: Critical Fix For: 1.4.0.SP1 The log of the experiment. * Empty database mysql> CONNECT ovidiu Connection id: 5288 Current database: ovidiu mysql> SHOW TABLES; Empty set (0.00 sec) mysql> * Both cluster nodes down. * Start node 39 2007-10-13 20:08:19,259 INFO [org.jboss.system.server.Server] JBoss (MX MicroKernel) [4.2.0.GA (build: SVNTag=JBPAPP_4_2_0_GA date=200706281411)] Started in 23s:774ms mysql> SELECT * FROM JBM_MSG_REF; Empty set (0.00 sec) mysql> SELECT * FROM JBM_POSTOFFICE; +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | JMS post office | 39 | bounceQueue | queue.bounceQueue | NULL | 7 | Y | N | | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N | | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N | ... | JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N | ... | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ 17 rows in set (0.00 sec) * Start node 38 2007-10-13 20:11:58,280 INFO [org.jboss.system.server.Server] JBoss (MX MicroKernel) [4.2.0.GA (build: SVNTag=JBPAPP_4_2_0_GA date=200706281411)] Started in 27s:214ms mysql> SELECT * FROM JBM_MSG_REF; Empty set (0.00 sec) mysql> SELECT * FROM JBM_POSTOFFICE; +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N | | JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N | ... | JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N | | JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N | | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N | | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N | | JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N | ... | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ 34 rows in set (0.00 sec) * Make sure I look up a clustered but non load-balanced connection factory from node 39 ofeodorov@yamato$ mess ping ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}] Destination: JBossQueue[ovidiu] node 39 alive closing connection JBossConnection->ConnectionDelegate[1542500044, ID=t2-w06lzq7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ... * Sending 50,000 10K persistent text messages to "ovidiu" on node 39. ofeodorov@yamato$ mess -stat send -size 10240 50000 ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}] Destination: JBossQueue[ovidiu] sent 38 message(s) with a rate of 37.9 messages/sec, 38 message(s) sent so far sent 32 message(s) with a rate of 32.0 messages/sec, 70 message(s) sent so far sent 33 message(s) with a rate of 33.0 messages/sec, 103 message(s) sent so far ..... sent 58 message(s) with a rate of 58.0 messages/sec, 49907 message(s) sent so far sent 41 message(s) with a rate of 41.0 messages/sec, 49948 message(s) sent so far 50000 message(s) sent non-transactionally at a rate of 59.1 messages/sec closing connection JBossConnection->ConnectionDelegate[1839586123, ID=03-t8gmzq7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ... * Database content after send: mysql> SELECT COUNT(MESSAGE_ID) FROM JBM_MSG_REF; +-------------------+ | COUNT(MESSAGE_ID) | +-------------------+ | 50000 | +-------------------+ mysql> SELECT MIN(MESSAGE_ID) FROM JBM_MSG_REF; +-----------------+ | MIN(MESSAGE_ID) | +-----------------+ | 0 | +-----------------+ mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 0; +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | 0 | 0 | NULL | C | 39070331482505216 | NULL | 0 | 0 | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ mysql> SELECT MAX(MESSAGE_ID) FROM JBM_MSG_REF; +-----------------+ | MAX(MESSAGE_ID) | +-----------------+ | 54192 | +-----------------+ mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 54192; +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | 0 | 54192 | NULL | C | 39070359199138639 | NULL | 0 | 0 | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ mysql> SELECT * FROM JBM_POSTOFFICE; +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N | | JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N | ... | JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N | ... | JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N | ... | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N | | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N | ... | JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N | ... | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ 34 rows in set (0.00 sec) * Receiving on 50 concurrent threads, from node 39 ofeodorov@yamato$ mess -stat -sessions 50 receive all ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}] Destination: JBossQueue[ovidiu] Consumer43 starting to consume, 1 active consumers ... Consumer06 starting to consume, 2 active consumers ... Consumer38 starting to consume, 3 active consumers ... Consumer13 starting to consume, 4 active consumers ... Consumer37 starting to consume, 5 active consumers ... Consumer10 starting to consume, 6 active consumers ... Consumer11 starting to consume, 7 active consumers ... Consumer25 starting to consume, 8 active consumers ... Consumer12 starting to consume, 9 active consumers ... Consumer04 starting to consume, 10 active consumers ... Consumer48 starting to consume, 11 active consumers ... Consumer32 starting to consume, 12 active consumers ... Consumer40 starting to consume, 13 active consumers ... Consumer39 starting to consume, 14 active consumers ... Consumer05 starting to consume, 15 active consumers ... Consumer09 starting to consume, 16 active consumers ... Consumer03 starting to consume, 17 active consumers ... Consumer41 starting to consume, 18 active consumers ... Consumer19 starting to consume, 19 active consumers ... Consumer07 starting to consume, 20 active consumers ... Consumer22 starting to consume, 21 active consumers ... Consumer36 starting to consume, 22 active consumers ... Consumer01 starting to consume, 23 active consumers ... Consumer46 starting to consume, 24 active consumers ... Consumer16 starting to consume, 25 active consumers ... Consumer18 starting to consume, 26 active consumers ... Consumer23 starting to consume, 28 active consumers ... Consumer33 starting to consume, 27 active consumers ... Consumer02 starting to consume, 29 active consumers ... Consumer00 starting to consume, 30 active consumers ... Consumer17 starting to consume, 31 active consumers ... Consumer15 starting to consume, 32 active consumers ... Consumer47 starting to consume, 33 active consumers ... Consumer31 starting to consume, 34 active consumers ... Consumer27 starting to consume, 35 active consumers ... Consumer49 starting to consume, 36 active consumers ... Consumer44 starting to consume, 37 active consumers ... Consumer26 starting to consume, 38 active consumers ... Consumer24 starting to consume, 39 active consumers ... Consumer42 starting to consume, 40 active consumers ... Consumer28 starting to consume, 41 active consumers ... Consumer20 starting to consume, 42 active consumers ... Consumer29 starting to consume, 43 active consumers ... Consumer35 starting to consume, 44 active consumers ... Consumer45 starting to consume, 45 active consumers ... Consumer30 starting to consume, 47 active consumers ... Consumer08 starting to consume, 46 active consumers ... received 240 message(s) with a rate of 239.3 messages/sec, 240 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 240 message(s) received so far received 1378 message(s) with a rate of 1,378.0 messages/sec, 1618 message(s) received so far received 451 message(s) with a rate of 451.0 messages/sec, 2069 message(s) received so far received 61 message(s) with a rate of 61.0 messages/sec, 2130 message(s) received so far received 348 message(s) with a rate of 348.0 messages/sec, 2478 message(s) received so far received 62 message(s) with a rate of 62.0 messages/sec, 2540 message(s) received so far received 958 message(s) with a rate of 957.0 messages/sec, 3498 message(s) received so far received 88 message(s) with a rate of 88.2 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far received 312 message(s) with a rate of 312.3 messages/sec, 3898 message(s) received so far received 681 message(s) with a rate of 681.0 messages/sec, 4579 message(s) received so far received 273 message(s) with a rate of 273.3 messages/sec, 4852 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4852 message(s) received so far received 2 message(s) with a rate of 2.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far received 153 message(s) with a rate of 153.0 messages/sec, 5007 message(s) received so far received 1126 message(s) with a rate of 1,124.9 messages/sec, 6133 message(s) received so far received 864 message(s) with a rate of 864.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far received 42 message(s) with a rate of 42.0 messages/sec, 7039 message(s) received so far received 441 message(s) with a rate of 441.4 messages/sec, 7480 message(s) received so far received 1803 message(s) with a rate of 1,803.0 messages/sec, 9283 message(s) received so far received 505 message(s) with a rate of 505.0 messages/sec, 9788 message(s) received so far received 974 message(s) with a rate of 975.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far received 597 message(s) with a rate of 597.6 messages/sec, 11359 message(s) received so far received 447 message(s) with a rate of 447.4 messages/sec, 11806 message(s) received so far received 783 message(s) with a rate of 783.0 messages/sec, 12589 message(s) received so far received 262 message(s) with a rate of 261.7 messages/sec, 12851 message(s) received so far received 267 message(s) with a rate of 267.0 messages/sec, 13118 message(s) received so far received 293 message(s) with a rate of 293.0 messages/sec, 13411 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far received 197 message(s) with a rate of 196.8 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far received 201 message(s) with a rate of 200.6 messages/sec, 13809 message(s) received so far received 757 message(s) with a rate of 757.8 messages/sec, 14566 message(s) received so far received 580 message(s) with a rate of 580.0 messages/sec, 15146 message(s) received so far received 683 message(s) with a rate of 683.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far closing connection JBossConnection->ConnectionDelegate[375652924, ID=83-8dse0r7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ... received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far received 40 message(s) with a rate of 40.0 messages/sec, 15869 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15869 message(s) received so far received 0 message(s) with a rate of 0.0 messages/sec, 15869 message(s) received so far Could not just close() connection, I had to kill the java process SELECT COUNT(MESSAGE_ID) FROM JBM_MSG_REF; +-------------------+ | COUNT(MESSAGE_ID) | +-------------------+ | 34091 | +-------------------+ 50000 - 34091 - 15869 = 40 messages unnacounted for (probably the client send the acknowledgment, but it did not actually receive it) * Tried to create a new connection to the server, the JNDI invocation hung. * Killed both servers, restarted. Database situation: mysql> SELECT COUNT(MESSAGE_ID) FROM JBM_MSG; +-------------------+ | COUNT(MESSAGE_ID) | +-------------------+ | 34091 | +-------------------+ mysql> SELECT MIN(MESSAGE_ID) FROM JBM_MSG_REF; +-----------------+ | MIN(MESSAGE_ID) | +-----------------+ | 8957 | +-----------------+ mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 8957; +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | 0 | 8957 | NULL | C | 39070335505044212 | NULL | 0 | 0 | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ mysql> SELECT MAX(MESSAGE_ID) FROM JBM_MSG_REF; +-----------------+ | MAX(MESSAGE_ID) | +-----------------+ | 54192 | +-----------------+ mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 54192; +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ | 0 | 54192 | NULL | C | 39070359199138639 | NULL | 0 | 0 | +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+ mysql> SELECT * FROM JBM_POSTOFFICE; +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ | JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N | | JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N | ... | JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N | ... | JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N | ... | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N | | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N | ... | JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 40 | Y | N | ... | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N | +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+ 34 rows in set (0.00 sec) * Pinged 39 ofeodorov@yamato$ mess ping ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}] Destination: JBossQueue[ovidiu] node 39 alive closing connection JBossConnection->ConnectionDelegate[735699586, ID=t2-b4k51r7f-1-h2w31r7f-q9dxny-31184a, SID=39] on exit ... * Browsed the queue (I don't expect any, since channel 0 does not exist anymore) ofeodorov@yamato$ mess browse ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}] Destination: JBossQueue[ovidiu] NO messages in queue closing connection JBossConnection->ConnectionDelegate[2002021820, ID=03-fah61r7f-1-h2w31r7f-q9dxny-31184a, SID=39] on exit ...