[infinispan-dev] DIST.retrieveFromRemoteSource

[Bela Ban]

On 2/5/12 9:44 AM, Dan Berindei wrote:
> On Sat, Feb 4, 2012 at 5:23 PM, Bela Ban<bban at redhat.com>  wrote:
>> No, Socket.send() is not a hotspot in the Java code.
>>
>> On the networking level, with UDP datagrams, a packet is placed into a
>> buffer and send() returns after the packet has been placed into the
>> buffer successfully. Some network stacks simply discard the packet if
>> the buffer is full, I assume.
>>
>> With TCP, send() blocks until the packet has been placed into the buffer
>> and an ack has been received. If the receiver can't keep up with the
>> sending rate, it'll reduce the window to 0, so send() blocks until the
>> receiver catches up.
>>
>> So if you send 2 requests in paralle over TCP, both threads would
>> block() on the send.
>>
>> So I don't think parallelization would make sense here, unless of course
>> you're doing something else, such as serialization, lock acquisition etc...
>>
>
> You're probably right...
>
> When I sent the email I had only seen that DataGramSocket.send takes
> 0.4% of the get worker's total time (which is huge considering that
> 99.3% is spent waiting for the remote node's response). I didn't
> notice that it's synchronized and the actual sending only takes half
> of that - sending the messages in parallel would create more lock
> contention on the socket *and* in JGroups.
>
> On the other hand, we're sending the messages to two different nodes,
> on two different sockets, so sending in parallel *may* improve the
> response time in scenarios with few worker threads. Certainly not
> worth making our heavy-load performance worse, but I thought it was
> worth mentioning.


If you think this makes a difference, why don't you make a temporary 
code change and measure its effect on performance ?


> Anyway, my primary motivation for this question was that I believe we
> could use GroupRequest instead of our FutureCollator to send our
> commands in parallel. They both do essentially the same thing, except
> FutureCollator has and extra indirection layer because it uses
> UnicastRequest.


Ah, ok. Yes, I think that 1 GroupRequest of 2 might be more efficient 
than 2 UnicastRequests for sending an 'anycast' message to 2 members. 
Perhaps the marshalling is done only once (I'd have to confirm that 
though) and we're only creating 1 data structure and add it to a hashmap 
(request/response correlation)... Certainly worth giving a try...


>  If there is any performance discrepancy at the moment
> between GroupRequest and FutureCollator+UnicastRequest, I don't think
> there is any fundamental reason why we can't "fix" GroupRequest to be
> just as efficient as FutureCollator+UnicastRequest.


You're assuming that FutureCollator+UnicastRequest is faster than 
GroupRequest, what are you basing your assumption on ? As I outlined 
above, I'd rather assume the opposite, although I don't know FutureCollator.

Maybe we should have a chat next week to go through the code that sends 
an anycast to 2 members, wdyt ?


> I think I just saw one such fix: in RequestCorrelator.sendRequest, if
> anycasting is enabled then it's making a copy of the buffer for each
> target member. I don't think that is necessary at all, in fact I think
> it should reuse both the buffer and the headers and only change the
> destination address.

No, this *is* necessary, I don't just make copies because I think this 
is fun !! :-)

Remember that a message might be retransmitted, so it is placed into a 
retransmit buffer. If M1 has destination A and M2 has destination B, and 
we send M1 first (to A), then change M1's destination to B, and send it, 
everything is fine. However, if we later get a retransmit request from 
B, we'd resend the message to A instead ! This is just 1 example, 
modifications of headers is another one.

Note that the copy does *not* copy the buffer (payload) itself, but only 
references it, so this is fast. Of course, nobody is supposed to modify 
the contents of the buffer itself...

-- 
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat