[jboss-jira] [JBoss JIRA] Commented: (JGRP-464) Credit request storms from flow control protocols

[Brian Stansberry (JIRA)]

    [ http://jira.jboss.com/jira/browse/JGRP-464?page=comments#action_12359491 ] 
            
Brian Stansberry commented on JGRP-464:
---------------------------------------

A couple notes on this:

1) The tests I mentioned were of http session replication where the client drivers were capable of sending 500 concurrent requests per server, and the Tomcat thread pool was large enough to accomodate that.  6 servers in the cluster, so overall the cluster would be servicing 3000 clients. The intent of the test was to deliberately overload the system for sustained periods.

Effect of this is you could have hundreds of threads blocking in FC.handleDownMessage().  This was tested with Branch_2_4, which uses concurrent.jar's ReentrantLock, which is not fair.  So, it's possible for a thread to be waiting for credit, and credit is received, but other more recently arrived threads consume all the credits.  So, under sustained overload, threads timing out of the block wait were not uncommon. I would see dozens of these threads time out from blocking and in series reacquire the lock and send a credit request, one after the other, all in short order.

2) I was wrong about NAKACK ignoring headers in its max_xmit_size check. TBH, I don't know why I saw the UDP errors. :(  The error occurred very consistently with Branch_2_4 code.  The error message was an IOException indicating a packet was too large.  In any case, once I prevented the "credit request storms" I no longer saw the errors.

> Credit request storms from flow control protocols
> -------------------------------------------------
>
>                 Key: JGRP-464
>                 URL: http://jira.jboss.com/jira/browse/JGRP-464
>             Project: JGroups
>          Issue Type: Bug
>    Affects Versions: 2.5
>            Reporter: Brian Stansberry
>         Assigned To: Bela Ban
>            Priority: Critical
>             Fix For: 2.5
>
>
> If an application has a set of threads that are trying to send more messages than the consumers can handle, request threads block in FC or SFC waiting for credits.  If they wait too long, they wake up and send a message requesting credit.  With the tests I'm running, large numbers of threads would block, and then one after another time out, wake up and ask for credit, allow within a very short period.  Basically spamming the cluster asking for credit (and getting credit back for each request). 
> That seems inefficient, but with SFC it was leading to error conditions.  Seems some other server's NAKACK requested retransmission of a set of messages, some or all of which were a large number of these "spam" credit requests.  The credit requests basically had no message body, just headers, so the NAKACK.max_xmit_size check wasn't assigning them any weight.  Effect was the resulting retransmission message was > 64K and couldn't be sent, resulting in an ERROR in UDP.
> Possible solution is to add a min_credit_request_interval such that a thread that wakes up from blocking will not request credit it another thread has already done so within the configured time.  For reference, ,my port of SFC to Branch_4_2 has an implementation of that concept.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira