[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1112) Highly concurrent clients lock a JBM server node and apparently cause persistent messages to "dissapear"
Ovidiu Feodorov (JIRA)
jira-events at lists.jboss.org
Sun Oct 14 13:26:03 EDT 2007
[ http://jira.jboss.com/jira/browse/JBMESSAGING-1112?page=comments#action_12382347 ]
Ovidiu Feodorov commented on JBMESSAGING-1112:
----------------------------------------------
You don't need to see any source code, you only need to read what I dumped above. Here, to make it easier:
mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 8957;
+------------+------------+
| CHANNEL_ID | MESSAGE_ID | ..........
+------------+------------+
| 0 | 8957 | ..........
+------------+------------+
mysql> SELECT * FROM JBM_POSTOFFICE;
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N |
| JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 40 | Y | N |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> Highly concurrent clients lock a JBM server node and apparently cause persistent messages to "dissapear"
> --------------------------------------------------------------------------------------------------------
>
> Key: JBMESSAGING-1112
> URL: http://jira.jboss.com/jira/browse/JBMESSAGING-1112
> Project: JBoss Messaging
> Issue Type: Bug
> Affects Versions: 1.4.0.GA
> Reporter: Ovidiu Feodorov
> Assigned To: Tim Fox
> Priority: Critical
> Fix For: Unscheduled
>
>
> The log of the experiment.
> * Empty database
> mysql> CONNECT ovidiu
> Connection id: 5288
> Current database: ovidiu
> mysql> SHOW TABLES;
> Empty set (0.00 sec)
> mysql>
> * Both cluster nodes down.
> * Start node 39
> 2007-10-13 20:08:19,259 INFO [org.jboss.system.server.Server] JBoss (MX MicroKernel) [4.2.0.GA (build: SVNTag=JBPAPP_4_2_0_GA date=200706281411)] Started in 23s:774ms
> mysql> SELECT * FROM JBM_MSG_REF;
> Empty set (0.00 sec)
> mysql> SELECT * FROM JBM_POSTOFFICE;
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> | JMS post office | 39 | bounceQueue | queue.bounceQueue | NULL | 7 | Y | N |
> | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N |
> | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N |
> ...
> | JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N |
> ...
> | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N |
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> 17 rows in set (0.00 sec)
> * Start node 38
> 2007-10-13 20:11:58,280 INFO [org.jboss.system.server.Server] JBoss (MX MicroKernel) [4.2.0.GA (build: SVNTag=JBPAPP_4_2_0_GA date=200706281411)] Started in 27s:214ms
> mysql> SELECT * FROM JBM_MSG_REF;
> Empty set (0.00 sec)
> mysql> SELECT * FROM JBM_POSTOFFICE;
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> | JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N |
> | JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N |
> ...
> | JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N |
> | JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N |
> | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N |
> | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N |
> | JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N |
> ...
> | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N |
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> 34 rows in set (0.00 sec)
> * Make sure I look up a clustered but non load-balanced connection factory from node 39
> ofeodorov at yamato$ mess ping
> ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
> Destination: JBossQueue[ovidiu]
> node 39 alive
> closing connection JBossConnection->ConnectionDelegate[1542500044, ID=t2-w06lzq7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ...
> * Sending 50,000 10K persistent text messages to "ovidiu" on node 39.
> ofeodorov at yamato$ mess -stat send -size 10240 50000
> ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
> Destination: JBossQueue[ovidiu]
> sent 38 message(s) with a rate of 37.9 messages/sec, 38 message(s) sent so far
> sent 32 message(s) with a rate of 32.0 messages/sec, 70 message(s) sent so far
> sent 33 message(s) with a rate of 33.0 messages/sec, 103 message(s) sent so far
> .....
> sent 58 message(s) with a rate of 58.0 messages/sec, 49907 message(s) sent so far
> sent 41 message(s) with a rate of 41.0 messages/sec, 49948 message(s) sent so far
> 50000 message(s) sent non-transactionally at a rate of 59.1 messages/sec
> closing connection JBossConnection->ConnectionDelegate[1839586123, ID=03-t8gmzq7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ...
> * Database content after send:
> mysql> SELECT COUNT(MESSAGE_ID) FROM JBM_MSG_REF;
> +-------------------+
> | COUNT(MESSAGE_ID) |
> +-------------------+
> | 50000 |
> +-------------------+
> mysql> SELECT MIN(MESSAGE_ID) FROM JBM_MSG_REF;
> +-----------------+
> | MIN(MESSAGE_ID) |
> +-----------------+
> | 0 |
> +-----------------+
> mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 0;
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY |
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> | 0 | 0 | NULL | C | 39070331482505216 | NULL | 0 | 0 |
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> mysql> SELECT MAX(MESSAGE_ID) FROM JBM_MSG_REF;
> +-----------------+
> | MAX(MESSAGE_ID) |
> +-----------------+
> | 54192 |
> +-----------------+
> mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 54192;
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY |
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> | 0 | 54192 | NULL | C | 39070359199138639 | NULL | 0 | 0 |
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> mysql> SELECT * FROM JBM_POSTOFFICE;
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> | JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N |
> | JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N |
> ...
> | JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N |
> ...
> | JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N |
> ...
> | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N |
> | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N |
> ...
> | JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N |
> ...
> | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N |
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> 34 rows in set (0.00 sec)
> * Receiving on 50 concurrent threads, from node 39
> ofeodorov at yamato$ mess -stat -sessions 50 receive all
> ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
> Destination: JBossQueue[ovidiu]
> Consumer43 starting to consume, 1 active consumers ...
> Consumer06 starting to consume, 2 active consumers ...
> Consumer38 starting to consume, 3 active consumers ...
> Consumer13 starting to consume, 4 active consumers ...
> Consumer37 starting to consume, 5 active consumers ...
> Consumer10 starting to consume, 6 active consumers ...
> Consumer11 starting to consume, 7 active consumers ...
> Consumer25 starting to consume, 8 active consumers ...
> Consumer12 starting to consume, 9 active consumers ...
> Consumer04 starting to consume, 10 active consumers ...
> Consumer48 starting to consume, 11 active consumers ...
> Consumer32 starting to consume, 12 active consumers ...
> Consumer40 starting to consume, 13 active consumers ...
> Consumer39 starting to consume, 14 active consumers ...
> Consumer05 starting to consume, 15 active consumers ...
> Consumer09 starting to consume, 16 active consumers ...
> Consumer03 starting to consume, 17 active consumers ...
> Consumer41 starting to consume, 18 active consumers ...
> Consumer19 starting to consume, 19 active consumers ...
> Consumer07 starting to consume, 20 active consumers ...
> Consumer22 starting to consume, 21 active consumers ...
> Consumer36 starting to consume, 22 active consumers ...
> Consumer01 starting to consume, 23 active consumers ...
> Consumer46 starting to consume, 24 active consumers ...
> Consumer16 starting to consume, 25 active consumers ...
> Consumer18 starting to consume, 26 active consumers ...
> Consumer23 starting to consume, 28 active consumers ...
> Consumer33 starting to consume, 27 active consumers ...
> Consumer02 starting to consume, 29 active consumers ...
> Consumer00 starting to consume, 30 active consumers ...
> Consumer17 starting to consume, 31 active consumers ...
> Consumer15 starting to consume, 32 active consumers ...
> Consumer47 starting to consume, 33 active consumers ...
> Consumer31 starting to consume, 34 active consumers ...
> Consumer27 starting to consume, 35 active consumers ...
> Consumer49 starting to consume, 36 active consumers ...
> Consumer44 starting to consume, 37 active consumers ...
> Consumer26 starting to consume, 38 active consumers ...
> Consumer24 starting to consume, 39 active consumers ...
> Consumer42 starting to consume, 40 active consumers ...
> Consumer28 starting to consume, 41 active consumers ...
> Consumer20 starting to consume, 42 active consumers ...
> Consumer29 starting to consume, 43 active consumers ...
> Consumer35 starting to consume, 44 active consumers ...
> Consumer45 starting to consume, 45 active consumers ...
> Consumer30 starting to consume, 47 active consumers ...
> Consumer08 starting to consume, 46 active consumers ...
> received 240 message(s) with a rate of 239.3 messages/sec, 240 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 240 message(s) received so far
> received 1378 message(s) with a rate of 1,378.0 messages/sec, 1618 message(s) received so far
> received 451 message(s) with a rate of 451.0 messages/sec, 2069 message(s) received so far
> received 61 message(s) with a rate of 61.0 messages/sec, 2130 message(s) received so far
> received 348 message(s) with a rate of 348.0 messages/sec, 2478 message(s) received so far
> received 62 message(s) with a rate of 62.0 messages/sec, 2540 message(s) received so far
> received 958 message(s) with a rate of 957.0 messages/sec, 3498 message(s) received so far
> received 88 message(s) with a rate of 88.2 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
> received 312 message(s) with a rate of 312.3 messages/sec, 3898 message(s) received so far
> received 681 message(s) with a rate of 681.0 messages/sec, 4579 message(s) received so far
> received 273 message(s) with a rate of 273.3 messages/sec, 4852 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4852 message(s) received so far
> received 2 message(s) with a rate of 2.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
> received 153 message(s) with a rate of 153.0 messages/sec, 5007 message(s) received so far
> received 1126 message(s) with a rate of 1,124.9 messages/sec, 6133 message(s) received so far
> received 864 message(s) with a rate of 864.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
> received 42 message(s) with a rate of 42.0 messages/sec, 7039 message(s) received so far
> received 441 message(s) with a rate of 441.4 messages/sec, 7480 message(s) received so far
> received 1803 message(s) with a rate of 1,803.0 messages/sec, 9283 message(s) received so far
> received 505 message(s) with a rate of 505.0 messages/sec, 9788 message(s) received so far
> received 974 message(s) with a rate of 975.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
> received 597 message(s) with a rate of 597.6 messages/sec, 11359 message(s) received so far
> received 447 message(s) with a rate of 447.4 messages/sec, 11806 message(s) received so far
> received 783 message(s) with a rate of 783.0 messages/sec, 12589 message(s) received so far
> received 262 message(s) with a rate of 261.7 messages/sec, 12851 message(s) received so far
> received 267 message(s) with a rate of 267.0 messages/sec, 13118 message(s) received so far
> received 293 message(s) with a rate of 293.0 messages/sec, 13411 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
> received 197 message(s) with a rate of 196.8 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
> received 201 message(s) with a rate of 200.6 messages/sec, 13809 message(s) received so far
> received 757 message(s) with a rate of 757.8 messages/sec, 14566 message(s) received so far
> received 580 message(s) with a rate of 580.0 messages/sec, 15146 message(s) received so far
> received 683 message(s) with a rate of 683.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> closing connection JBossConnection->ConnectionDelegate[375652924, ID=83-8dse0r7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ...
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
> received 40 message(s) with a rate of 40.0 messages/sec, 15869 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15869 message(s) received so far
> received 0 message(s) with a rate of 0.0 messages/sec, 15869 message(s) received so far
> Could not just close() connection, I had to kill the java process
> SELECT COUNT(MESSAGE_ID) FROM JBM_MSG_REF;
> +-------------------+
> | COUNT(MESSAGE_ID) |
> +-------------------+
> | 34091 |
> +-------------------+
> 50000 - 34091 - 15869 = 40 messages unnacounted for (probably the client send the acknowledgment, but it did not actually receive it)
> * Tried to create a new connection to the server, the JNDI invocation hung.
> * Killed both servers, restarted.
> Database situation:
> mysql> SELECT COUNT(MESSAGE_ID) FROM JBM_MSG;
> +-------------------+
> | COUNT(MESSAGE_ID) |
> +-------------------+
> | 34091 |
> +-------------------+
> mysql> SELECT MIN(MESSAGE_ID) FROM JBM_MSG_REF;
> +-----------------+
> | MIN(MESSAGE_ID) |
> +-----------------+
> | 8957 |
> +-----------------+
> mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 8957;
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY |
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> | 0 | 8957 | NULL | C | 39070335505044212 | NULL | 0 | 0 |
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> mysql> SELECT MAX(MESSAGE_ID) FROM JBM_MSG_REF;
> +-----------------+
> | MAX(MESSAGE_ID) |
> +-----------------+
> | 54192 |
> +-----------------+
> mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 54192;
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> | CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY |
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> | 0 | 54192 | NULL | C | 39070359199138639 | NULL | 0 | 0 |
> +------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
> mysql> SELECT * FROM JBM_POSTOFFICE;
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> | POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> | JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N |
> | JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N |
> ...
> | JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N |
> ...
> | JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N |
> ...
> | JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N |
> | JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N |
> ...
> | JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 40 | Y | N |
> ...
> | JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N |
> +-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
> 34 rows in set (0.00 sec)
> * Pinged 39
> ofeodorov at yamato$ mess ping
> ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
> Destination: JBossQueue[ovidiu]
> node 39 alive
> closing connection JBossConnection->ConnectionDelegate[735699586, ID=t2-b4k51r7f-1-h2w31r7f-q9dxny-31184a, SID=39] on exit ...
> * Browsed the queue (I don't expect any, since channel 0 does not exist anymore)
> ofeodorov at yamato$ mess browse
> ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
> Destination: JBossQueue[ovidiu]
> NO messages in queue
> closing connection JBossConnection->ConnectionDelegate[2002021820, ID=03-fah61r7f-1-h2w31r7f-q9dxny-31184a, SID=39] on exit ...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list