The concurrent stack has done a great job in parallelizing delivery of messages from different senders, but is there a way to parallelize this from the same sender as well? Now I know this doesn't make sense in terms of JGroups messages (order needs to be maintained) but when you consider moving up one level of abstraction in JBC, this causes a bottleneck. Consider, for example: 1. Using sync replication and BR. 2. 3 threads on cache1, working on disjoint data sets (no contention at all, writes can be in parallel). 3. When it comes to replication to the same buddy, the concurrent stack on the receiver will queue the 3 commits. 4. All parallelization achieved in 2. above is wasted since the transactions need to queue anyway. Is there something we can do with custom message headers, etc. to allow for parallel delivery of these messages? E.g., perhaps using a "communication id", which could be a representation of a global transaction? Bela, Vladimir, any thoughts? Cheers, -- Manik Surtani Lead, JBoss Cache manik(a)jboss.org _______________________________________________ jbosscache-dev mailing list jbosscache-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbosscache-dev

I believe this should be possible if you use OOB messages and custom headers. Let's confirm with Bela. Manik Surtani wrote: > The concurrent stack has done a great job in parallelizing delivery > of messages from different senders, but is there a way to parallelize > this from the same sender as well? Now I know this doesn't make > sense in terms of JGroups messages (order needs to be maintained) but > when you consider moving up one level of abstraction in JBC, this > causes a bottleneck. Consider, for example: > > 1. Using sync replication and BR. > 2. 3 threads on cache1, working on disjoint data sets (no contention > at all, writes can be in parallel). > 3. When it comes to replication to the same buddy, the concurrent > stack on the receiver will queue the 3 commits. > 4. All parallelization achieved in 2. above is wasted since the > transactions need to queue anyway. > > Is there something we can do with custom message headers, etc. to > allow for parallel delivery of these messages? E.g., perhaps using a > "communication id", which could be a representation of a global > transaction? Bela, Vladimir, any thoughts? > > Cheers, > -- > Manik Surtani > Lead, JBoss Cache > manik(a)jboss.org > > > > > > > _______________________________________________ > jbosscache-dev mailing list > jbosscache-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jbosscache-dev _______________________________________________ jbosscache-dev mailing list jbosscache-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbosscache-dev

Manik Surtani wrote: > The concurrent stack has done a great job in parallelizing delivery > of messages from different senders, but is there a way to > parallelize this from the same sender as well? Now I know this > doesn't make sense in terms of JGroups messages (order needs to be > maintained) Not necessarily. Sometimes you just want reliable delivery but don't care about ordering. To do that and still provide once-only delivery, mark your message as OOB (out-of-band): Message msg=new Message(); msg.setFlag(Message.OOB); // msg.clearFlag(Message.OOB) clears this flag again OR use one of the RpcDispatcher.callRemoteMethod(s) methods with the OOB parameter set to true.

> but when you consider moving up one level of abstraction in JBC, > this causes a bottleneck. Consider, for example: > > 1. Using sync replication and BR. > 2. 3 threads on cache1, working on disjoint data sets (no > contention at all, writes can be in parallel). > 3. When it comes to replication to the same buddy, the concurrent > stack on the receiver will queue the 3 commits. > 4. All parallelization achieved in 2. above is wasted since the > transactions need to queue anyway. You can use the above mechanism. However, you have to make sure that any thread * uses synchronous replication * the commits are synchronous as well This won't work for asynchronous replication, but I don't think it is needed here anyway right ?

Manik Surtani wrote: > >> You can use the above mechanism. However, you have to make sure >> that any thread >> >> * uses synchronous replication >> * the commits are synchronous as well >> >> This won't work for asynchronous replication, but I don't think it >> is needed here anyway right ? > > It is most critical in sync replication, correct, but why should it > not work with the async case as well? I'm guessing misaligned > packets? Commit(A::TX-1) could be received *after* Commit(A::TX-2). Admitted, a rare case, but if the COMMIT(A:TX-1) is lost and needs to be retransmitted, this could happen

Manik Surtani wrote: > On 13 May 2008, at 14:22, Bela Ban wrote: >> >> >> Manik Surtani wrote: >>> >>>> You can use the above mechanism. However, you have to make sure >>>> that any thread >>>> >>>> * uses synchronous replication >>>> * the commits are synchronous as well >>>> >>>> This won't work for asynchronous replication, but I don't think >>>> it is needed here anyway right ? >>> >>> It is most critical in sync replication, correct, but why should >>> it not work with the async case as well? I'm guessing misaligned >>> packets? >> >> Commit(A::TX-1) could be received *after* Commit(A::TX-2). >> Admitted, a rare case, but if the COMMIT(A:TX-1) is lost and needs >> to be retransmitted, this could happen > The TxInterceptor would just throw out Commit(A::TX-2) if it hasn't > received a prepare for that TX yet. Locks would still be held by > TX1 assuming it's prepare has already been received. > I guess the problem is in 1-PC but that is to be expected anyway. It's not to be expected with properly ordered messages. TX1 is sent before TX2, just doesn't get received first (perhaps dropped by network and retransmitted). If you remove the message ordering guarantee and don't have some sort of ack like REPL_SYNC gives you, transactions can be received in arbitrary order.

[Semi-OT] On his last post on the forum, fungrim said the locking issue he was seeing was on the sender side, not the receiver. This discussion is about removing contention on the receiver. I'm *speculating* what he's seeing is blocking in FC.