[jboss-jira] [JBoss JIRA] Commented: (JGRP-376) TLS: use priority based queue rather than BoundedLinkedQueue for default thread pool
[ http://jira.jboss.com/jira/browse/JGRP-376?page=comments#action_12348537 ]
Bela Ban commented on JGRP-376:
-------------------------------
Observation:
We want to run messages from the *same* sender on the same thread, we do *not* want to
process messages P1, P2, P3 and P4 (msgs 1-4 from P) each on its separate thread ! Why ?
Because, if we assume we have 4 threads T1-4 processing P1-4, T1-4 will block on the
NakReceiverWindow for P in NAKACK, and only *one* thread will be allowed to pass its
message up (in strict FIFO order for P). 3 threads will block on the same window, and when
finally able to access the window, all 3 remaining threads will terminate, because the
first thread most likely already passed all 4 messages up the stack !
SOLUTION:
- Create a thread pool that's twice the number of nodes in the cluster, adjust it
dynamically based on
membership changes
- Create a priority queue (BoundedChannel impl, as suggested in this JIRA issue), which
- has 1 queue for each unicast sender
- has 1 queue per multicast sender
- maintains a total number of bytes (all queues accumulated), which cannot exceed
max_bytes
- whenever a message is added, we block unless the message's length doesn't
exceed max_bytes
- whenever a message is removed, we decrement the total number of bytes and possibly
unblock adders
- each message from P goes into a queue for P
- whenever a message is removed from a queue by a worker in the thread pool, we
increment a counter
for that queue
- whenever a worker is done, we decrement that counter. So the counter for P's queue
essentially
tells us how many messages are currently processed for P
- on a take() or poll(), we try to find a queue whose counter is 0
- if no queue has its counter == 0, we can always fall back to one of the schemes
described above, e.g.
fullest queue first, round robin etc
If we set the thread pool's max size to be twice the size of the cluster, then, if
every node is sending unicast and multicast messages, we ensure that only 1 thread is
processing unicast or mcast messages for a given node P.
TLS: use priority based queue rather than BoundedLinkedQueue for
default thread pool
------------------------------------------------------------------------------------
Key: JGRP-376
URL: http://jira.jboss.com/jira/browse/JGRP-376
Project: JGroups
Issue Type: Feature Request
Affects Versions: 2.4
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.5
This will allow us to dispatch incoming messages to threads from the pool based on
criteria.
- The impl implements interface BoundedChannel
- It has N queues, 1 for each sender (possibly those are BoundedLinkedQueues)
- There is a max size in bytes for the priority queue, each internal queue has max_size /
N bytes where N = number of members. This is
dynamically adjusted
ADDITION
- An internal queue maintains the number of bytes (Message.length()), an offer() or put()
increases that amount by the size of the message, a
take() or poll() decreases the amount. When a put() or offer() would exceed the max
amount, it will block until a take() or poll() decreases
it such that the new message can be added to the queue. We might also discard messages
to full queues, or implement somthing akin to
RED (random early detection), which starts discarding messages *before* the queue is
full
REMOVAL
- Strategy pluggable, e.g.
- Take from the fullest queue
- Take from the least full queue
- Round robin
- Weighted round robin (weighted by capacity of each queue), e.g. take relative to
capacity from each queue. Example:
Queues A (2MB), B (2MB), C (6MB): we take 2 from A, 2 from B and 6 from C
- Random robin
etc
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira