Question about per-socket concurrency

[Yang Zhang]

Hi, we have a Netty-based server that is running into a bottleneck that 
we suspect may be due to the way (socket-wise) concurrency control works 
in the Netty reactor.  While we're exploring the code, we're wondering 
if anyone has any insight into this so as to expedite our performance 
debugging.

The system is a simple topic-based publish-subscribe messaging system, 
and the workload we're issuing is a handful of publishers publishing 
messages to the server, all on separate hosts on a GigE LAN.  Each of 
the publishers tops out at just 1000 messages per second, where each 
message is 1KB in size.  However, we can keep piling on clients, and the 
throughput scales up linearly.

 From this info, one simple explanation would be that the culprit is a 
bottleneck in the client.  Yet the strange thing is that the CPU 
utilization of each client is just ~5%.  On the server, CPU utilization 
hovers at ~20% when presented with a single publisher, and grows another 
~20% for each additional publisher.  The bottom line is that we're being 
held back well before full CPU or network saturation.

Is there any synchronization in the reactor core of Netty that could be 
causing this per-socket bottleneck?  Thanks in advance for any hints.
-- 
Yang Zhang
http://www.mit.edu/~y_z/