[jboss-jira] [JBoss JIRA] Commented: (JGRP-376) TLS: use priority based queue rather than BoundedLinkedQueue for default thread pool

[Bela Ban (JIRA)]

    [ http://jira.jboss.com/jira/browse/JGRP-376?page=comments#action_12348537 ] 
            
Bela Ban commented on JGRP-376:
-------------------------------

Observation:
We want to run messages from the *same* sender on the same thread, we do *not* want to process messages P1, P2, P3 and P4 (msgs 1-4 from P) each on its separate thread ! Why ? Because, if we assume we have 4 threads T1-4 processing P1-4, T1-4 will block on the NakReceiverWindow for P in NAKACK, and only *one* thread will be allowed to pass its message up (in strict FIFO order for P). 3 threads will block on the same window, and when finally able to access the window, all 3 remaining threads will terminate, because the first thread most likely already passed all 4 messages up the stack !

SOLUTION:
- Create a thread pool that's twice the number of nodes in the cluster, adjust it dynamically based on 
   membership changes
- Create a priority queue (BoundedChannel impl, as suggested in this JIRA issue), which
  - has 1 queue for each unicast sender
  - has 1 queue per multicast sender
  - maintains a total number of bytes (all queues accumulated), which cannot exceed max_bytes
  - whenever a message is added, we block unless the message's length doesn't exceed max_bytes
  - whenever a message is removed, we decrement the total number of bytes and possibly unblock adders
  - each message from P goes into a queue for P
  - whenever a message is removed from a queue by a worker in the thread pool, we increment a counter
    for that queue
  - whenever a worker is done, we decrement that counter. So the counter for P's queue essentially 
    tells us how many messages are currently processed for P
  - on a take() or poll(), we try to find a queue whose counter is 0
  - if no queue has its counter == 0, we can always fall back to one of the schemes described above, e.g.
    fullest queue first, round robin etc

If we set the thread pool's max size to be twice the size of the cluster, then, if every node is sending unicast and multicast messages, we ensure that only 1 thread is processing unicast or mcast messages for a given node P.




> TLS: use priority based queue rather than BoundedLinkedQueue for default thread pool
> ------------------------------------------------------------------------------------
>
>                 Key: JGRP-376
>                 URL: http://jira.jboss.com/jira/browse/JGRP-376
>             Project: JGroups
>          Issue Type: Feature Request
>    Affects Versions: 2.4
>            Reporter: Bela Ban
>         Assigned To: Bela Ban
>             Fix For: 2.5
>
>
> This will allow us to dispatch incoming messages to threads from the pool based on criteria.
> - The impl implements interface BoundedChannel
> - It has N queues, 1 for each sender (possibly those are BoundedLinkedQueues)
> - There is a max size in bytes for the priority queue, each internal queue has max_size / N bytes where N = number of members. This is
>   dynamically adjusted
> ADDITION
> - An internal queue maintains the number of bytes (Message.length()), an offer() or put() increases that amount by the size of the message, a
>   take() or poll() decreases the amount. When a put() or offer() would exceed the max amount, it will block until a take() or poll() decreases   
>   it such that the new message can be added to the queue. We might also discard messages to full queues, or implement somthing akin to
>   RED (random early detection), which starts discarding messages *before* the queue is full
> REMOVAL
> - Strategy pluggable, e.g.
>   - Take from the fullest queue
>   - Take from the least full queue
>   - Round robin
>   - Weighted round robin (weighted by capacity of each queue), e.g. take relative to capacity from each queue. Example:
>      Queues A (2MB), B (2MB), C (6MB): we take 2 from A, 2 from B and 6 from C
>   - Random robin
> etc

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira