[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-984) Expiring messages hang JBoss server and client

[Gabriele Garuglieri (JIRA)]

    [ http://jira.jboss.com/jira/browse/JBMESSAGING-984?page=comments#action_12364840 ] 
            
Gabriele Garuglieri commented on JBMESSAGING-984:
-------------------------------------------------

I forgot to mention that the deadlock problem could be not strictly related to expiration.
I'm planning to use JBmessaging with an application that should handle tenth of subscribers and hundredth millions messages a day so i'm trying to stress a bit it to see its reliability.
It happens, way too often, that suddendly server and client freeze to the point that to be able to receive or send any other message or open a new connection i have to kill both client and server. And sometimes this is not enough, i even had cases where i had to clean jbm_msg and jbm_msg_ref before restarting the server, to be able to open new connections.
The problem is that this seems to be happening randomly and i could never be able to find any relation to time of activity, number of messages or an exact sequence of actions to reproduce the deadlock until i tried to use expiring messages as described above.
This same problem was present in 1.2.0, 1.2.0.sp1, and now in 1.3.0.GA.

> Expiring messages hang JBoss server and client
> ----------------------------------------------
>
>                 Key: JBMESSAGING-984
>                 URL: http://jira.jboss.com/jira/browse/JBMESSAGING-984
>             Project: JBoss Messaging
>          Issue Type: Bug
>    Affects Versions: 1.3.0.GA
>         Environment: JBoss 4.2.0.GA, JBmessaging 1.3.0.GA, Java 1.6.0_01, Windows XP Pro sp2, Postgres 8.2.4
>            Reporter: Gabriele Garuglieri
>         Assigned To: Tim Fox
>            Priority: Blocker
>         Attachments: JBMESSAGING-984.zip
>
>
> I think i've found a serious bug in messaging behaviour.
> The problem is always reproducible with the following steps:
> 1- send a couple thousands of messages with a short expiration time to a queue (i used 20 sec to shorten the test time, the problem does not depend on expiration time value). I used the /queue/testQueue
> 2- wait enough for the messages to expire
> 3- start a client that read from /queue/testQueue
> 4- the client won't receive any message because as messaging is getting messages from channel 2 it expires them and updates them to channel 1
> 5- the problem is that it never gets through, after expiring approximately 1025 messages (not exactly the same number in each run) everything gets stuck
> The client locks in Connection.close(), that never returns (even if i kill the server!!!).
> JBoss server locks who knows where, such that if i try to orderly shutdown the server, it won't terminate until i kill it, or if i try to open a new connection from a client to server mostly of the times it will hang in ConnectionFactory.createConnection(), that will never return.
> The run for which i'm sending doc was done whith a fresh db.
> Included you'll find:
> server boot.log and server.log, 
> client thread dump while it was locked in Connection.close(),
> server thread dump while it was locked after shutdown request,
> csv dump of jbm_counter, jbm_msg, jbm_msg_ref, jbm_postoffice taken after all those weird things happened
> Thread dumps of client and server were taken with jstack utility since none of them were responding to ctrl-break.
> The messaging configuration was based on default server and all the attributes were left at their default.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira