[jboss-jira] [JBoss JIRA] Created: (JBMESSAGING-1112) Highly concurrent clients lock a JBM server node and apparently cause persistent messages to "dissapear"
Ovidiu Feodorov (JIRA)
jira-events at lists.jboss.org
Sun Oct 14 00:11:12 EDT 2007
Highly concurrent clients lock a JBM server node and apparently cause persistent messages to "dissapear"
--------------------------------------------------------------------------------------------------------
Key: JBMESSAGING-1112
URL: http://jira.jboss.com/jira/browse/JBMESSAGING-1112
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 1.4.0.GA
Reporter: Ovidiu Feodorov
Assigned To: Tim Fox
Priority: Critical
The log of the experiment.
* Empty database
mysql> CONNECT ovidiu
Connection id: 5288
Current database: ovidiu
mysql> SHOW TABLES;
Empty set (0.00 sec)
mysql>
* Both cluster nodes down.
* Start node 39
2007-10-13 20:08:19,259 INFO [org.jboss.system.server.Server] JBoss (MX MicroKernel) [4.2.0.GA (build: SVNTag=JBPAPP_4_2_0_GA date=200706281411)] Started in 23s:774ms
mysql> SELECT * FROM JBM_MSG_REF;
Empty set (0.00 sec)
mysql> SELECT * FROM JBM_POSTOFFICE;
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| JMS post office | 39 | bounceQueue | queue.bounceQueue | NULL | 7 | Y | N |
| JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N |
| JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N |
...
| JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N |
...
| JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
17 rows in set (0.00 sec)
* Start node 38
2007-10-13 20:11:58,280 INFO [org.jboss.system.server.Server] JBoss (MX MicroKernel) [4.2.0.GA (build: SVNTag=JBPAPP_4_2_0_GA date=200706281411)] Started in 27s:214ms
mysql> SELECT * FROM JBM_MSG_REF;
Empty set (0.00 sec)
mysql> SELECT * FROM JBM_POSTOFFICE;
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N |
| JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N |
...
| JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N |
| JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N |
| JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N |
| JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N |
| JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N |
...
| JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
34 rows in set (0.00 sec)
* Make sure I look up a clustered but non load-balanced connection factory from node 39
ofeodorov at yamato$ mess ping
ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
Destination: JBossQueue[ovidiu]
node 39 alive
closing connection JBossConnection->ConnectionDelegate[1542500044, ID=t2-w06lzq7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ...
* Sending 50,000 10K persistent text messages to "ovidiu" on node 39.
ofeodorov at yamato$ mess -stat send -size 10240 50000
ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
Destination: JBossQueue[ovidiu]
sent 38 message(s) with a rate of 37.9 messages/sec, 38 message(s) sent so far
sent 32 message(s) with a rate of 32.0 messages/sec, 70 message(s) sent so far
sent 33 message(s) with a rate of 33.0 messages/sec, 103 message(s) sent so far
.....
sent 58 message(s) with a rate of 58.0 messages/sec, 49907 message(s) sent so far
sent 41 message(s) with a rate of 41.0 messages/sec, 49948 message(s) sent so far
50000 message(s) sent non-transactionally at a rate of 59.1 messages/sec
closing connection JBossConnection->ConnectionDelegate[1839586123, ID=03-t8gmzq7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ...
* Database content after send:
mysql> SELECT COUNT(MESSAGE_ID) FROM JBM_MSG_REF;
+-------------------+
| COUNT(MESSAGE_ID) |
+-------------------+
| 50000 |
+-------------------+
mysql> SELECT MIN(MESSAGE_ID) FROM JBM_MSG_REF;
+-----------------+
| MIN(MESSAGE_ID) |
+-----------------+
| 0 |
+-----------------+
mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 0;
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
| CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY |
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
| 0 | 0 | NULL | C | 39070331482505216 | NULL | 0 | 0 |
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
mysql> SELECT MAX(MESSAGE_ID) FROM JBM_MSG_REF;
+-----------------+
| MAX(MESSAGE_ID) |
+-----------------+
| 54192 |
+-----------------+
mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 54192;
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
| CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY |
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
| 0 | 54192 | NULL | C | 39070359199138639 | NULL | 0 | 0 |
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
mysql> SELECT * FROM JBM_POSTOFFICE;
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N |
| JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N |
...
| JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N |
...
| JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N |
...
| JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N |
| JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N |
...
| JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 0 | Y | N |
...
| JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
34 rows in set (0.00 sec)
* Receiving on 50 concurrent threads, from node 39
ofeodorov at yamato$ mess -stat -sessions 50 receive all
ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
Destination: JBossQueue[ovidiu]
Consumer43 starting to consume, 1 active consumers ...
Consumer06 starting to consume, 2 active consumers ...
Consumer38 starting to consume, 3 active consumers ...
Consumer13 starting to consume, 4 active consumers ...
Consumer37 starting to consume, 5 active consumers ...
Consumer10 starting to consume, 6 active consumers ...
Consumer11 starting to consume, 7 active consumers ...
Consumer25 starting to consume, 8 active consumers ...
Consumer12 starting to consume, 9 active consumers ...
Consumer04 starting to consume, 10 active consumers ...
Consumer48 starting to consume, 11 active consumers ...
Consumer32 starting to consume, 12 active consumers ...
Consumer40 starting to consume, 13 active consumers ...
Consumer39 starting to consume, 14 active consumers ...
Consumer05 starting to consume, 15 active consumers ...
Consumer09 starting to consume, 16 active consumers ...
Consumer03 starting to consume, 17 active consumers ...
Consumer41 starting to consume, 18 active consumers ...
Consumer19 starting to consume, 19 active consumers ...
Consumer07 starting to consume, 20 active consumers ...
Consumer22 starting to consume, 21 active consumers ...
Consumer36 starting to consume, 22 active consumers ...
Consumer01 starting to consume, 23 active consumers ...
Consumer46 starting to consume, 24 active consumers ...
Consumer16 starting to consume, 25 active consumers ...
Consumer18 starting to consume, 26 active consumers ...
Consumer23 starting to consume, 28 active consumers ...
Consumer33 starting to consume, 27 active consumers ...
Consumer02 starting to consume, 29 active consumers ...
Consumer00 starting to consume, 30 active consumers ...
Consumer17 starting to consume, 31 active consumers ...
Consumer15 starting to consume, 32 active consumers ...
Consumer47 starting to consume, 33 active consumers ...
Consumer31 starting to consume, 34 active consumers ...
Consumer27 starting to consume, 35 active consumers ...
Consumer49 starting to consume, 36 active consumers ...
Consumer44 starting to consume, 37 active consumers ...
Consumer26 starting to consume, 38 active consumers ...
Consumer24 starting to consume, 39 active consumers ...
Consumer42 starting to consume, 40 active consumers ...
Consumer28 starting to consume, 41 active consumers ...
Consumer20 starting to consume, 42 active consumers ...
Consumer29 starting to consume, 43 active consumers ...
Consumer35 starting to consume, 44 active consumers ...
Consumer45 starting to consume, 45 active consumers ...
Consumer30 starting to consume, 47 active consumers ...
Consumer08 starting to consume, 46 active consumers ...
received 240 message(s) with a rate of 239.3 messages/sec, 240 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 240 message(s) received so far
received 1378 message(s) with a rate of 1,378.0 messages/sec, 1618 message(s) received so far
received 451 message(s) with a rate of 451.0 messages/sec, 2069 message(s) received so far
received 61 message(s) with a rate of 61.0 messages/sec, 2130 message(s) received so far
received 348 message(s) with a rate of 348.0 messages/sec, 2478 message(s) received so far
received 62 message(s) with a rate of 62.0 messages/sec, 2540 message(s) received so far
received 958 message(s) with a rate of 957.0 messages/sec, 3498 message(s) received so far
received 88 message(s) with a rate of 88.2 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 3586 message(s) received so far
received 312 message(s) with a rate of 312.3 messages/sec, 3898 message(s) received so far
received 681 message(s) with a rate of 681.0 messages/sec, 4579 message(s) received so far
received 273 message(s) with a rate of 273.3 messages/sec, 4852 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4852 message(s) received so far
received 2 message(s) with a rate of 2.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 4854 message(s) received so far
received 153 message(s) with a rate of 153.0 messages/sec, 5007 message(s) received so far
received 1126 message(s) with a rate of 1,124.9 messages/sec, 6133 message(s) received so far
received 864 message(s) with a rate of 864.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 6997 message(s) received so far
received 42 message(s) with a rate of 42.0 messages/sec, 7039 message(s) received so far
received 441 message(s) with a rate of 441.4 messages/sec, 7480 message(s) received so far
received 1803 message(s) with a rate of 1,803.0 messages/sec, 9283 message(s) received so far
received 505 message(s) with a rate of 505.0 messages/sec, 9788 message(s) received so far
received 974 message(s) with a rate of 975.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 10762 message(s) received so far
received 597 message(s) with a rate of 597.6 messages/sec, 11359 message(s) received so far
received 447 message(s) with a rate of 447.4 messages/sec, 11806 message(s) received so far
received 783 message(s) with a rate of 783.0 messages/sec, 12589 message(s) received so far
received 262 message(s) with a rate of 261.7 messages/sec, 12851 message(s) received so far
received 267 message(s) with a rate of 267.0 messages/sec, 13118 message(s) received so far
received 293 message(s) with a rate of 293.0 messages/sec, 13411 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13411 message(s) received so far
received 197 message(s) with a rate of 196.8 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 13608 message(s) received so far
received 201 message(s) with a rate of 200.6 messages/sec, 13809 message(s) received so far
received 757 message(s) with a rate of 757.8 messages/sec, 14566 message(s) received so far
received 580 message(s) with a rate of 580.0 messages/sec, 15146 message(s) received so far
received 683 message(s) with a rate of 683.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
closing connection JBossConnection->ConnectionDelegate[375652924, ID=83-8dse0r7f-1-qvc9zq7f-g0vqau-31184a, SID=39] on exit ...
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15829 message(s) received so far
received 40 message(s) with a rate of 40.0 messages/sec, 15869 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15869 message(s) received so far
received 0 message(s) with a rate of 0.0 messages/sec, 15869 message(s) received so far
Could not just close() connection, I had to kill the java process
SELECT COUNT(MESSAGE_ID) FROM JBM_MSG_REF;
+-------------------+
| COUNT(MESSAGE_ID) |
+-------------------+
| 34091 |
+-------------------+
50000 - 34091 - 15869 = 40 messages unnacounted for (probably the client send the acknowledgment, but it did not actually receive it)
* Tried to create a new connection to the server, the JNDI invocation hung.
* Killed both servers, restarted.
Database situation:
mysql> SELECT COUNT(MESSAGE_ID) FROM JBM_MSG;
+-------------------+
| COUNT(MESSAGE_ID) |
+-------------------+
| 34091 |
+-------------------+
mysql> SELECT MIN(MESSAGE_ID) FROM JBM_MSG_REF;
+-----------------+
| MIN(MESSAGE_ID) |
+-----------------+
| 8957 |
+-----------------+
mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 8957;
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
| CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY |
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
| 0 | 8957 | NULL | C | 39070335505044212 | NULL | 0 | 0 |
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
mysql> SELECT MAX(MESSAGE_ID) FROM JBM_MSG_REF;
+-----------------+
| MAX(MESSAGE_ID) |
+-----------------+
| 54192 |
+-----------------+
mysql> SELECT * FROM JBM_MSG_REF WHERE MESSAGE_ID = 54192;
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
| CHANNEL_ID | MESSAGE_ID | TRANSACTION_ID | STATE | ORD | PAGE_ORD | DELIVERY_COUNT | SCHED_DELIVERY |
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
| 0 | 54192 | NULL | C | 39070359199138639 | NULL | 0 | 0 |
+------------+------------+----------------+-------+-------------------+----------+----------------+----------------+
mysql> SELECT * FROM JBM_POSTOFFICE;
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED | ALL_NODES |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
| JMS post office | 38 | DLQ | queue.DLQ | NULL | 21 | N | N |
| JMS post office | 38 | ExpiryQueue | queue.ExpiryQueue | NULL | 22 | N | N |
...
| JMS post office | 38 | ovidiu | queue.ovidiu | NULL | 20 | Y | N |
...
| JMS post office | 38 | testDistributedQueue | queue.testDistributedQueue | NULL | 28 | Y | N |
...
| JMS post office | 39 | DLQ | queue.DLQ | NULL | 1 | N | N |
| JMS post office | 39 | ExpiryQueue | queue.ExpiryQueue | NULL | 2 | N | N |
...
| JMS post office | 39 | ovidiu | queue.ovidiu | NULL | 40 | Y | N |
...
| JMS post office | 39 | testDistributedQueue | queue.testDistributedQueue | NULL | 8 | Y | N |
+-----------------+---------+--------------------------+--------------------------------+----------+------------+-----------+-----------+
34 rows in set (0.00 sec)
* Pinged 39
ofeodorov at yamato$ mess ping
ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
Destination: JBossQueue[ovidiu]
node 39 alive
closing connection JBossConnection->ConnectionDelegate[735699586, ID=t2-b4k51r7f-1-h2w31r7f-q9dxny-31184a, SID=39] on exit ...
* Browsed the queue (I don't expect any, since channel 0 does not exist anymore)
ofeodorov at yamato$ mess browse
ConnectionFactory: JBossConnectionFactory->ClusteredConnectionFactoryDelegate[SIDs={38,39}]
Destination: JBossQueue[ovidiu]
NO messages in queue
closing connection JBossConnection->ConnectionDelegate[2002021820, ID=03-fah61r7f-1-h2w31r7f-q9dxny-31184a, SID=39] on exit ...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list