Bela Ban created JGRP-2143: ------------------------------ Summary: TP: use only one thread per member to pass up regular messages Key: JGRP-2143 URL: https://issues.jboss.org/browse/JGRP-2143 Project: JGroups Issue Type: Enhancement Reporter: Bela Ban Assignee: Bela Ban Fix For: 4.0 This applies only to _regular_ messages; OOB and internal messages are processed by passing them to the thread pool directly when they've been received. The processing of a message received from B is as follows: * A regular message (or message batch) is assigned a thread from the thread pool and passed up to the reliability protocol, e.g. NAKACK2 or UNICAST3. * There is is added to the table for B. * The thread sees if another thread is already delivering messages from B to the application. If not, it grabs as many consecutive (ordered) messages from the table and delivers them to the application. Otherwise, it returns and can be assigned other tasks. The problem here is that more than one thread may be passing up messages from a given sender B; only at the NAKACK2 or UNICAST3 level will a single thread be selected to deliver the messages to the application. This causes higher thread pool usage than required, with all of its drawbacks, e.g. more context switching, higher contention on adding messages to the table for B, and possibly exhaustion of the thread pool. An example of where service is denied or delayed: * We have a cluster of {A,B,C,D} * A receives 10 messages from B, 4 from C and 1 from D * The thread pool's max size is 20 * The 10 messages from B are processed; all 10 threads add their messages to the table, but only 1 delivers them to the application and the other 9 return to the pool * 4 messages from C are added to C's table, 1 thread delivers them and 3 return * The 1 message from D is added to D's table and the same thread is used to deliver the message up the stack to the application So while we receive 15 messages, effectively only 3 threads are needed to deliver them to the application: as these are regular messages, they need to be delivered in _sender order_. The 9 threads which process messages from B are only adding them to B's table and then return immediately. This causes increased context switching, plus more contention on B's table (which is synchronized) -- This message was sent by Atlassian JIRA (v7.2.3#72005)