On 19 Mar 2013, at 15:07, Sanne Grinovero <sanne(a)redhat.com&gt; wrote: > > > ----- Original Message ----- >> >> On 19 Mar 2013, at 12:21, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >> >>> On 19 Mar 2013, at 11:05, Sanne Grinovero wrote: >>>> Does Marshalling really need to be performed in a separate thread >>>> pool? >>>> I think we have too many pools, too much context switching, and >>>> situations like this one which should be avoided. >>>> >>>> We could document it but all these details are making it very >>>> hard to feel comfortable with, and for this specific use case I >>>> wonder if there >>>> is a strong benefit: plain serial operations seem so much cleaner >>>> to me. >>> +1 for dropping it in 6.0. It isn't enabled by default and AFAIK it >>> created more confusion through the users than benefits. >> >> Why? I don't agree. If network transfer is the most expensive part >> of performing a write, then marshalling is the second-most >> expensive. If you don't take the marshalling offline as well, >> you're only realising a part of the performance gains of using >> async. > > Of course. I didn't mean to put it on the thread of the invoker, I would expect > this to happen "behind the scenes" when using async, but in the same thread which > is managing the lower IO so to reduce both context switching and these weird > race conditions.. so removing the option only. Well, when using the same lower IO pool, while common sense, isn't as easy since it is a JGroups pool. If we pass the marshaller itself into JGroups, the marshalling still happens online, and just the IO happening in a separate thread. Also, JGroups allows you to register one marshaller and unmarshaller per channel - which doesn't work when you have a transport shared by multiple cache instances potentially on different class loaders. So yes, this can be done much better, but that means a fair few changes in JGroups such that: * Marshalling happens in the async thread (the same one that puts the message on the wire) rather than in the caller's thread * sendMessage() should accept a marshaller and unmarshaller per invocation Then we can drop this additional thread pool.

> >> >>> On top of that the number of pools is growing (5.3 adds another >>> pool in the scope of ISPN-2808). >> >> You can configure to use a single thread pool for all these tasks, if >> hanging on to multiple thread pools is too complex. > > I don't believe you can always do that, if you don't keep tasks isolated > in different pools deadlocks could happen. So unless you can come up with > a nice diagram and explain which ones are safe to share, it is very > complex to handle. >

> Would be nice to have these discussions on the public mailing list. +1. Adding infinispan-dev in cc. > > Sanne > >> >> - M >> >> -- >> Manik Surtani >> manik(a)jboss.org >> twitter.com/maniksurtani >> >> Platform Architect, JBoss Data Grid >> http://red.ht/data-grid >> >> >> > -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 19 Mar 2013, at 15:07, Sanne Grinovero <sanne(a)redhat.com&gt; wrote: > > > ----- Original Message ----- >> >> On 19 Mar 2013, at 12:21, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >> >>> On 19 Mar 2013, at 11:05, Sanne Grinovero wrote: >>>> Does Marshalling really need to be performed in a separate thread >>>> pool? >>>> I think we have too many pools, too much context switching, and >>>> situations like this one which should be avoided. >>>> >>>> We could document it but all these details are making it very >>>> hard to feel comfortable with, and for this specific use case I >>>> wonder if there >>>> is a strong benefit: plain serial operations seem so much cleaner >>>> to me. >>> +1 for dropping it in 6.0. It isn't enabled by default and AFAIK it >>> created more confusion through the users than benefits. >> >> Why? I don't agree. If network transfer is the most expensive part >> of performing a write, then marshalling is the second-most >> expensive. If you don't take the marshalling offline as well, >> you're only realising a part of the performance gains of using >> async. > > Of course. I didn't mean to put it on the thread of the invoker, I would expect > this to happen "behind the scenes" when using async, but in the same thread which > is managing the lower IO so to reduce both context switching and these weird > race conditions.. so removing the option only. Well, when using the same lower IO pool, while common sense, isn't as easy since it is a JGroups pool. If we pass the marshaller itself into JGroups, the marshalling still happens online, and just the IO happening in a separate thread. Also, JGroups allows you to register one marshaller and unmarshaller per channel - which doesn't work when you have a transport shared by multiple cache instances potentially on different class loaders. So yes, this can be done much better, but that means a fair few changes in JGroups such that: * Marshalling happens in the async thread (the same one that puts the message on the wire) rather than in the caller's thread

* sendMessage() should accept a marshaller and unmarshaller per invocation

Then we can drop this additional thread pool. > >> >>> On top of that the number of pools is growing (5.3 adds another >>> pool in the scope of ISPN-2808). >> >> You can configure to use a single thread pool for all these tasks, if >> hanging on to multiple thread pools is too complex. > > I don't believe you can always do that, if you don't keep tasks isolated > in different pools deadlocks could happen. So unless you can come up with > a nice diagram and explain which ones are safe to share, it is very > complex to handle. > > Would be nice to have these discussions on the public mailing list. +1. Adding infinispan-dev in cc. > > Sanne > >> >> - M >> >> -- >> Manik Surtani >> manik(a)jboss.org >> twitter.com/maniksurtani >> >> Platform Architect, JBoss Data Grid >> http://red.ht/data-grid >> >> >> > -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> * Marshalling happens in the async thread (the same one that puts the message on the wire) rather than in the caller's thread my understanding is that there's no such additional thread, but caller's thread goes to the network stack even for async calls. I think Bela can put some light on this. If you use bundling, the bundler thread writes the bytes to the socket, not the caller's thread. TCP also uses a special sender thread for each connection, if use_sender_queues=true (the default).

> * sendMessage() should accept a marshaller and unmarshaller per invocation There is a org.jgroups.Buffer that we pass to the org.jgroups.Message we send across, another, less intrusive way would be to write a lazy wrapper around it. Still, FRAG2 needs the actual bytes of the message, and that's a long time before the message gets passed to the bundler thread.

Cheers Dan _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev