[infinispan-dev] Minutes Infinispan meeting in London

[Bela Ban]

Mircea, Dan, Pedro, Sanne and I had a meeting in London this week on how 
to use the new features of JGroups 3.3 in Infinispan 5.3, I've copied 
the minutes from the wiki below.


  London meeting

Bela, Pedro, Mircea, Dan, Sanne


      Message bundling and OOB messages

In 3.3, all messages will be bundled, not just regular messages, but

also OOB messages. The way this works on the sender side is:

- A thread sending a message in the transport adds it to a queue

- There's one thread which dequeues messages and sends them as bundles

   - It sends a messages bundle if the max size has been reached, or

     there are no more messages in the queue

   - This means single messages are sent immediately, or we fill up a

     bundle (in a few microseconds) and send it

Impact on Infinispan:

- Use DONT_BUNDLE instead of OOB if you don't want to bundle messages

   - However, even DONT_BUNDLE might get deprecated

- If we have 1 sender invoking sync RPCs, we don't need to set

   DONT_BUNDLE anymore

- If we have multiple senders invoking sync RPCs, performance should

   get better as RPCs and responses are bundled

- Since bundling will result in message *batches* on the receiver,

   performance should increase in general


      Message batching

Message bundles sent by a sender are received as message batches

(MessageBatch) by the receivers. When a batch is received, the batch

is passed up using up(MessageBatch).

Protocols can remove / replace / add messages in a batch and pass the

batch further up.

The advantage of a batch is that resources such as locks are acquired

only once for a batch of N messages rather than N times. Example: when

NAKACK2 receives a batch of 10 messages, it adds the 10 messages to

the receiver table in a bulk operation, which is more efficient than

doing this 10 times.

Further optimizations on batching (probably 3.4):

- Remove similar ops, e.g. UNICAST3 acks for A:15, A:25 and A:35 can

   be clubbed together into just ack(A:35)

- Merge similar headers, e.g. multicast messages 20-30 can be orderd

   by seqno, and we simply send a range [20..30] and let the receiver

   generate the headers on the fly


      Async Invocation API (AIA)

JGroups only passes up messages to Infinispan, which then uses its own

thread pool to deliver them. E.g. based on Pedro's code for TO, we

could parallelize delivery based on the target keys of the

transaction. E.g if we have tx1 modifying keys {A,B,C} and tx2

modifying keys {T,U}, then tx1 and tx2 can be run concurrently.

If tx1 and tx2 modify overlapping key sets, then tx2 would be queued

and executed *after* tx1, not taking up a thread from the pool,

reducing the chances of the thread pool maxing out and also

ensuring different threads are not going to contend on the locks

on same keys.

The implementation could be done in an interceptor fronting the 
interceptor stack, which queues dependent TXs and

- when ready to be executed - sends them up the interceptor stack on a 
thread from

the internal pool.

Infinispan having its own thread pool means that JGroups threads will

not block anymore, e.g. trying to acquire a lock for a TX. The size of

those pools can therefore be reduced.

The advantage of AIA is that it's up to Infinispan, not JGroups, how

to deliver messages. JGroups delivers messages based on the order in

which they were sent by a sender (FIFO), whereas Infinispan can make

much more informed decisions as to how to deliver the messages.


      Internal thread pool for JGroups

All JGroups internal messages use the internal thread pool (message

flag=INTERNAL). Not having to share the OOB pool with apps (such as

Infinispan) means that internal messages can always be processed, and

are not discarded or blocked, e.g. by a maxed out thread pool.

The internal pool can be switched off, and - if AIA is implemented in

Infinispan - the number of OOB and regular threads can be massively

reduced. The internal thread pool doesn't need to be big either.


      UNICAST3

Successor to UNICAST and UNICAST2, best of both worlds. Acks single

messages quickly, so we have no first-msg-lost or last-msg-lost issues

anymore. Doesn't generate many acks though.

Proposed to trigger and ACK only after a certain number of messages

rather than after any batch to avoid ACK on small batches.

https://issues.jboss.org/browse/JGRP-1594

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)