[infinispan-dev] Minutes Infinispan meeting in London

Mircea, Dan, Pedro, Sanne and I had a meeting in London this week on how to use the new features of JGroups 3.3 in Infinispan 5.3, I've copied the minutes from the wiki below. London meeting Bela, Pedro, Mircea, Dan, Sanne Message bundling and OOB messages In 3.3, all messages will be bundled, not just regular messages, but also OOB messages. The way this works on the sender side is: - A thread sending a message in the transport adds it to a queue - There's one thread which dequeues messages and sends them as bundles - It sends a messages bundle if the max size has been reached, or there are no more messages in the queue - This means single messages are sent immediately, or we fill up a bundle (in a few microseconds) and send it Impact on Infinispan: - Use DONT_BUNDLE instead of OOB if you don't want to bundle messages - However, even DONT_BUNDLE might get deprecated - If we have 1 sender invoking sync RPCs, we don't need to set DONT_BUNDLE anymore - If we have multiple senders invoking sync RPCs, performance should get better as RPCs and responses are bundled - Since bundling will result in message *batches* on the receiver, performance should increase in general Message batching Message bundles sent by a sender are received as message batches (MessageBatch) by the receivers. When a batch is received, the batch is passed up using up(MessageBatch). Protocols can remove / replace / add messages in a batch and pass the batch further up. The advantage of a batch is that resources such as locks are acquired only once for a batch of N messages rather than N times. Example: when NAKACK2 receives a batch of 10 messages, it adds the 10 messages to the receiver table in a bulk operation, which is more efficient than doing this 10 times. Further optimizations on batching (probably 3.4): - Remove similar ops, e.g. UNICAST3 acks for A:15, A:25 and A:35 can be clubbed together into just ack(A:35) - Merge similar headers, e.g. multicast messages 20-30 can be orderd by seqno, and we simply send a range [20..30] and let the receiver generate the headers on the fly Async Invocation API (AIA) JGroups only passes up messages to Infinispan, which then uses its own thread pool to deliver them. E.g. based on Pedro's code for TO, we could parallelize delivery based on the target keys of the transaction. E.g if we have tx1 modifying keys {A,B,C} and tx2 modifying keys {T,U}, then tx1 and tx2 can be run concurrently. If tx1 and tx2 modify overlapping key sets, then tx2 would be queued and executed *after* tx1, not taking up a thread from the pool, reducing the chances of the thread pool maxing out and also ensuring different threads are not going to contend on the locks on same keys. The implementation could be done in an interceptor fronting the interceptor stack, which queues dependent TXs and - when ready to be executed - sends them up the interceptor stack on a thread from the internal pool. Infinispan having its own thread pool means that JGroups threads will not block anymore, e.g. trying to acquire a lock for a TX. The size of those pools can therefore be reduced. The advantage of AIA is that it's up to Infinispan, not JGroups, how to deliver messages. JGroups delivers messages based on the order in which they were sent by a sender (FIFO), whereas Infinispan can make much more informed decisions as to how to deliver the messages. Internal thread pool for JGroups All JGroups internal messages use the internal thread pool (message flag=INTERNAL). Not having to share the OOB pool with apps (such as Infinispan) means that internal messages can always be processed, and are not discarded or blocked, e.g. by a maxed out thread pool. The internal pool can be switched off, and - if AIA is implemented in Infinispan - the number of OOB and regular threads can be massively reduced. The internal thread pool doesn't need to be big either. UNICAST3 Successor to UNICAST and UNICAST2, best of both worlds. Acks single messages quickly, so we have no first-msg-lost or last-msg-lost issues anymore. Doesn't generate many acks though. Proposed to trigger and ACK only after a certain number of messages rather than after any batch to avoid ACK on small batches. https://issues.jboss.org/browse/JGRP-1594 -- Bela Ban, JGroups lead (http://www.jgroups.org)