Hello, As many of you should know, I'm refactoring the remoting package. I was trying and testing what can be done and it's time for an update. Finally, I got a working version (needs some cleanups) that can be found here [1]. The goal is to reduce the complexity from CommandAwareRpcDispatcher and InboundInvocationHandler. This classes are getting full of "if command is type T, then do this" and it will get worse when ISPN-2849 [2] and ISPN-4610 [3] is done. Also, it decouples the Transport from the processing remote commands logic. The main idea is to have one global inbound invocations handler and a per cache inbound invocations handler. The first one will handle the non-cache rpc commands and the later the cache rpc commands. Since each cache has a different configuration, multiple per cache inbound invocation handlers will be implemented. Currently, I have the non-total order and a total order implementation. After ISPN-2849 and ISPN-4610, I'll probably add more, for example: TO tx and TO non-tx, pessimistic, optimistic and default non-tx implementations. change details: * remove total order remote executor service. The remote executor service can be used and it has exactly the same goal. * added a single thread remote executor service. This will handle the FIFO deliver commands. Previously, they were handled by JGroups incoming threads and with a new executor service, each cache can process their own FIFO commands concurrently.

* currently the non-blocking cache rpc commands are processed directly in JGroups threads. Not sure if it is ok, but I think we can add another remote executor service for these commands. The advantage is that the JGroups threads no longer will execute remote commands. * the Transport is decoupled. May be useful for the test suite. * possibly remove the TotalOrder*Command (TotalOrderPrepareCommand and similar) (needs to double check). Since we have a total order inbound invocation handler for total order transactions, it is not necessary a special command to identify when the transaction is in total order or not. Comments, ideas, feedback are welcome. Cheers, Pedro [1] https://github.com/pruivo/infinispan/compare/remoting_refactor [2] https://issues.jboss.org/browse/ISPN-2849 => don't keep thread blocked waiting for locks [3] https://issues.jboss.org/browse/ISPN-4610 => non transactional total order _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 11/06/2014 03:01 PM, Bela Ban wrote: > > > On 06/11/14 15:36, Pedro Ruivo wrote: >> >> * added a single thread remote executor service. This will handle the >> FIFO deliver commands. Previously, they were handled by JGroups incoming >> threads and with a new executor service, each cache can process their >> own FIFO commands concurrently. > > +1000. This allows multiple updates from the same sender but to > different caches to be executed in parallel, and will speed thing up. > > Do you intend to share a thread pool between the invocations handlers of > the various caches, or do they each have their own thread pool ? Or is > this configurable ? > That is question that cross my mind and I don't have any idea what would be the best. So, for now, I will leave the thread pool shared between the handlers. Never thought to make it configurable, but maybe that is the best option. And maybe, it should be possible to have different max-thread size per cache. For example: * all caches using this remote executor will share the same instance <remote-executor name="shared" shared="true" max-threads=4.../> * all caches using this remote executor will create their own thread pool with max-threads equals to 1 <remote-executor name="low-throughput-cache" shared="false" max-threads=1 .../> * all caches using this remote executor will create their own thread pool with max-threads equals to 1000 <remote executor name="high-throughput-cache" shared="false" max-thread=1000 .../> is this what you have in mind? comments? Cheers, Pedro _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I second the opinion that any threadpools should be shared by default. There are users who have hundreds or thousands of caches and having separate threadpool for each of them could easily drain resources. And sharing resources is the purpose of threadpools, right?

My opinion is that we should aim for less configuration, i.e. threadpools should mostly have sensible defaults and be shared by default unless there are extremely good reasons for not doing so. Tristan On 06/11/14 19:40, Radim Vansa wrote: > I second the opinion that any threadpools should be shared by default. > There are users who have hundreds or thousands of caches and having > separate threadpool for each of them could easily drain resources. And > sharing resources is the purpose of threadpools, right? > > Radim > > On 11/06/2014 04:37 PM, Bela Ban wrote: >> #1 I would by default have 1 thread pool shared by all caches >> #2 This global thread pool should be configurable, perhaps in the >> <global> section ? >> #3 Each cache by default uses the gobal thread pool >> #4 A cache can define its own thread pool, then it would use this one >> and not the global thread pool >> >> I think this gives you a mixture between ease of use and flexibility in >> configuring pool per cache if needed >> >> On 06/11/14 16:23, Pedro Ruivo wrote: >>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>> * added a single thread remote executor service. This will handle the >>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>> threads and with a new executor service, each cache can process their >>>>> own FIFO commands concurrently. >>>> +1000. This allows multiple updates from the same sender but to >>>> different caches to be executed in parallel, and will speed thing up. >>>> >>>> Do you intend to share a thread pool between the invocations handlers of >>>> the various caches, or do they each have their own thread pool ? Or is >>>> this configurable ? >>>> >>> That is question that cross my mind and I don't have any idea what would >>> be the best. So, for now, I will leave the thread pool shared between >>> the handlers. >>> >>> Never thought to make it configurable, but maybe that is the best >>> option. And maybe, it should be possible to have different max-thread >>> size per cache. For example: >>> >>> * all caches using this remote executor will share the same instance >>> <remote-executor name="shared" shared="true" max-threads=4.../> >>> >>> * all caches using this remote executor will create their own thread >>> pool with max-threads equals to 1 >>> <remote-executor name="low-throughput-cache" shared="false" >>> max-threads=1 .../> >>> >>> * all caches using this remote executor will create their own thread >>> pool with max-threads equals to 1000 >>> <remote executor name="high-throughput-cache" shared="false" >>> max-thread=1000 .../> >>> >>> is this what you have in mind? comments? >>> >>> Cheers, >>> Pedro >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Btw., have you ever considered checks if a thread returns to pool reasonably often? Some of the other datagrids use this, though there's not much how to react upon that beyond printing out stack traces (but you can at least report to management that some node seems to be broken). Radim On 11/07/2014 08:35 AM, Bela Ban wrote: > That's exactly what I suggested. No config gives you a shared global > thread pool for all caches. > > Those caches which need a separate pool can do that via configuration > (and of course also programmatically) > > On 06/11/14 20:31, Tristan Tarrant wrote: >> My opinion is that we should aim for less configuration, i.e. >> threadpools should mostly have sensible defaults and be shared by >> default unless there are extremely good reasons for not doing so. >> >> Tristan >> >> On 06/11/14 19:40, Radim Vansa wrote: >>> I second the opinion that any threadpools should be shared by default. >>> There are users who have hundreds or thousands of caches and having >>> separate threadpool for each of them could easily drain resources. And >>> sharing resources is the purpose of threadpools, right? >>> >>> Radim >>> >>> On 11/06/2014 04:37 PM, Bela Ban wrote: >>>> #1 I would by default have 1 thread pool shared by all caches >>>> #2 This global thread pool should be configurable, perhaps in the >>>> <global> section ? >>>> #3 Each cache by default uses the gobal thread pool >>>> #4 A cache can define its own thread pool, then it would use this one >>>> and not the global thread pool >>>> >>>> I think this gives you a mixture between ease of use and flexibility in >>>> configuring pool per cache if needed >>>> >>>> On 06/11/14 16:23, Pedro Ruivo wrote: >>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>>>> * added a single thread remote executor service. This will handle the >>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>>>> threads and with a new executor service, each cache can process their >>>>>>> own FIFO commands concurrently. >>>>>> +1000. This allows multiple updates from the same sender but to >>>>>> different caches to be executed in parallel, and will speed thing up. >>>>>> >>>>>> Do you intend to share a thread pool between the invocations handlers of >>>>>> the various caches, or do they each have their own thread pool ? Or is >>>>>> this configurable ? >>>>>> >>>>> That is question that cross my mind and I don't have any idea what would >>>>> be the best. So, for now, I will leave the thread pool shared between >>>>> the handlers. >>>>> >>>>> Never thought to make it configurable, but maybe that is the best >>>>> option. And maybe, it should be possible to have different max-thread >>>>> size per cache. For example: >>>>> >>>>> * all caches using this remote executor will share the same instance >>>>> <remote-executor name="shared" shared="true" max-threads=4.../> >>>>> >>>>> * all caches using this remote executor will create their own thread >>>>> pool with max-threads equals to 1 >>>>> <remote-executor name="low-throughput-cache" shared="false" >>>>> max-threads=1 .../> >>>>> >>>>> * all caches using this remote executor will create their own thread >>>>> pool with max-threads equals to 1000 >>>>> <remote executor name="high-throughput-cache" shared="false" >>>>> max-thread=1000 .../> >>>>> >>>>> is this what you have in mind? comments? >>>>> >>>>> Cheers, >>>>> Pedro >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>

I think our priority should be to get rid of the need for threadpools - not their configuration options. If there is a real need for threadpools, then you have to provide full configuration options as you simply don't know how it's going to be used nor on what kind of hardware it's going to be run.

Sounds like yet another reason to see if we should split the configuration in two areas: - high level simple configuration (what you need to set to get started) - expert tuning (what you'll need when production time comes)

Also some of most recent users on Hibernate forums are puzzled on how to do tuning for Infinispan, when they're deploying several applications using it in the same container. I'm educating them on FORK, but we should be able to go beyond that: allow containers and platform developers to share threadpools across CacheManagers, so in such a case you'd want Infinispan to use a service for threadpool management, and allow people to inject a custom component for it.

Sanne On 7 November 2014 12:21, Bela Ban <bban(a)redhat.com&gt; wrote: > Hi Radim, > > no I haven't. However, you can replace the thread pools used by JGroups > and use custom pools. > > I like another idea better: inject Byteman code at runtime that keeps > track of this, and *other useful stats as well*. > > It would be very useful to support if we could ship a package to a > customer that is injected into their running system and grabs all the > vital stats we need for a few minutes, then removes itself again and > those stats are then sent to use as a ZIP file. > The good thing about byteman is that it can remove itself without a > trace; ie. there's no overhead before / after running byteman. > > > On 07/11/14 09:31, Radim Vansa wrote: >> Btw., have you ever considered checks if a thread returns to pool >> reasonably often? Some of the other datagrids use this, though there's >> not much how to react upon that beyond printing out stack traces (but >> you can at least report to management that some node seems to be broken). >> >> Radim >> >> On 11/07/2014 08:35 AM, Bela Ban wrote: >>> That's exactly what I suggested. No config gives you a shared global >>> thread pool for all caches. >>> >>> Those caches which need a separate pool can do that via configuration >>> (and of course also programmatically) >>> >>> On 06/11/14 20:31, Tristan Tarrant wrote: >>>> My opinion is that we should aim for less configuration, i.e. >>>> threadpools should mostly have sensible defaults and be shared by >>>> default unless there are extremely good reasons for not doing so. >>>> >>>> Tristan >>>> >>>> On 06/11/14 19:40, Radim Vansa wrote: >>>>> I second the opinion that any threadpools should be shared by default. >>>>> There are users who have hundreds or thousands of caches and having >>>>> separate threadpool for each of them could easily drain resources. And >>>>> sharing resources is the purpose of threadpools, right? >>>>> >>>>> Radim >>>>> >>>>> On 11/06/2014 04:37 PM, Bela Ban wrote: >>>>>> #1 I would by default have 1 thread pool shared by all caches >>>>>> #2 This global thread pool should be configurable, perhaps in the >>>>>> <global> section ? >>>>>> #3 Each cache by default uses the gobal thread pool >>>>>> #4 A cache can define its own thread pool, then it would use this one >>>>>> and not the global thread pool >>>>>> >>>>>> I think this gives you a mixture between ease of use and flexibility in >>>>>> configuring pool per cache if needed >>>>>> >>>>>> On 06/11/14 16:23, Pedro Ruivo wrote: >>>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>>>>>> * added a single thread remote executor service. This will handle the >>>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>>>>>> threads and with a new executor service, each cache can process their >>>>>>>>> own FIFO commands concurrently. >>>>>>>> +1000. This allows multiple updates from the same sender but to >>>>>>>> different caches to be executed in parallel, and will speed thing up. >>>>>>>> >>>>>>>> Do you intend to share a thread pool between the invocations handlers of >>>>>>>> the various caches, or do they each have their own thread pool ? Or is >>>>>>>> this configurable ? >>>>>>>> >>>>>>> That is question that cross my mind and I don't have any idea what would >>>>>>> be the best. So, for now, I will leave the thread pool shared between >>>>>>> the handlers. >>>>>>> >>>>>>> Never thought to make it configurable, but maybe that is the best >>>>>>> option. And maybe, it should be possible to have different max-thread >>>>>>> size per cache. For example: >>>>>>> >>>>>>> * all caches using this remote executor will share the same instance >>>>>>> <remote-executor name="shared" shared="true" max-threads=4.../> >>>>>>> >>>>>>> * all caches using this remote executor will create their own thread >>>>>>> pool with max-threads equals to 1 >>>>>>> <remote-executor name="low-throughput-cache" shared="false" >>>>>>> max-threads=1 .../> >>>>>>> >>>>>>> * all caches using this remote executor will create their own thread >>>>>>> pool with max-threads equals to 1000 >>>>>>> <remote executor name="high-throughput-cache" shared="false" >>>>>>> max-thread=1000 .../> >>>>>>> >>>>>>> is this what you have in mind? comments? >>>>>>> >>>>>>> Cheers, >>>>>>> Pedro >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >> >> > > -- > Bela Ban, JGroups lead (http://www.jgroups.org) > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hijacking thread 'Remoting package refactor' as the discussion has shifted. Sure, AOP is another approach. However, besided another limitations, Byteman rules are quite fragile with respect to different versions: if you're injecting code based on internal implementation method, when the name/signature changes, the rule is broken. Sometimes you even have to use AT LINE to formulate the injection point.

Would you accept a compile-time dependency to some annotations package in JGroups that could 'tag' the injection points? The idea is that anyone changing the source code would move the injection point annotations as well.

I was already thinking about this in relation with Message Flow Tracer [1] (not working right now as the JGroups have changed since I was writing that)?

Roman Macor is right now updating the rules and I was hoping that we could insert annotations into JGroups that would be used instead of the rules (I was already considering different AOP framework as Byteman does not allow AT EXIT to catch on leaving exceptions [2]).

Radim [1] https://github.com/rvansa/message-flow-tracer [2] https://issues.jboss.org/browse/BYTEMAN-237 On 11/07/2014 01:21 PM, Bela Ban wrote: > Hi Radim, > > no I haven't. However, you can replace the thread pools used by JGroups > and use custom pools. > > I like another idea better: inject Byteman code at runtime that keeps > track of this, and *other useful stats as well*. > > It would be very useful to support if we could ship a package to a > customer that is injected into their running system and grabs all the > vital stats we need for a few minutes, then removes itself again and > those stats are then sent to use as a ZIP file. > The good thing about byteman is that it can remove itself without a > trace; ie. there's no overhead before / after running byteman. > > > On 07/11/14 09:31, Radim Vansa wrote: >> Btw., have you ever considered checks if a thread returns to pool >> reasonably often? Some of the other datagrids use this, though there's >> not much how to react upon that beyond printing out stack traces (but >> you can at least report to management that some node seems to be broken). >> >> Radim >> >> On 11/07/2014 08:35 AM, Bela Ban wrote: >>> That's exactly what I suggested. No config gives you a shared global >>> thread pool for all caches. >>> >>> Those caches which need a separate pool can do that via configuration >>> (and of course also programmatically) >>> >>> On 06/11/14 20:31, Tristan Tarrant wrote: >>>> My opinion is that we should aim for less configuration, i.e. >>>> threadpools should mostly have sensible defaults and be shared by >>>> default unless there are extremely good reasons for not doing so. >>>> >>>> Tristan >>>> >>>> On 06/11/14 19:40, Radim Vansa wrote: >>>>> I second the opinion that any threadpools should be shared by default. >>>>> There are users who have hundreds or thousands of caches and having >>>>> separate threadpool for each of them could easily drain resources. And >>>>> sharing resources is the purpose of threadpools, right? >>>>> >>>>> Radim >>>>> >>>>> On 11/06/2014 04:37 PM, Bela Ban wrote: >>>>>> #1 I would by default have 1 thread pool shared by all caches >>>>>> #2 This global thread pool should be configurable, perhaps in the >>>>>> <global> section ? >>>>>> #3 Each cache by default uses the gobal thread pool >>>>>> #4 A cache can define its own thread pool, then it would use this one >>>>>> and not the global thread pool >>>>>> >>>>>> I think this gives you a mixture between ease of use and flexibility in >>>>>> configuring pool per cache if needed >>>>>> >>>>>> On 06/11/14 16:23, Pedro Ruivo wrote: >>>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>>>>>> * added a single thread remote executor service. This will handle the >>>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>>>>>> threads and with a new executor service, each cache can process their >>>>>>>>> own FIFO commands concurrently. >>>>>>>> +1000. This allows multiple updates from the same sender but to >>>>>>>> different caches to be executed in parallel, and will speed thing up. >>>>>>>> >>>>>>>> Do you intend to share a thread pool between the invocations handlers of >>>>>>>> the various caches, or do they each have their own thread pool ? Or is >>>>>>>> this configurable ? >>>>>>>> >>>>>>> That is question that cross my mind and I don't have any idea what would >>>>>>> be the best. So, for now, I will leave the thread pool shared between >>>>>>> the handlers. >>>>>>> >>>>>>> Never thought to make it configurable, but maybe that is the best >>>>>>> option. And maybe, it should be possible to have different max-thread >>>>>>> size per cache. For example: >>>>>>> >>>>>>> * all caches using this remote executor will share the same instance >>>>>>> <remote-executor name="shared" shared="true" max-threads=4.../> >>>>>>> >>>>>>> * all caches using this remote executor will create their own thread >>>>>>> pool with max-threads equals to 1 >>>>>>> <remote-executor name="low-throughput-cache" shared="false" >>>>>>> max-threads=1 .../> >>>>>>> >>>>>>> * all caches using this remote executor will create their own thread >>>>>>> pool with max-threads equals to 1000 >>>>>>> <remote executor name="high-throughput-cache" shared="false" >>>>>>> max-thread=1000 .../> >>>>>>> >>>>>>> is this what you have in mind? comments? >>>>>>> >>>>>>> Cheers, >>>>>>> Pedro >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>

On 11/07/2014 02:27 PM, Bela Ban wrote: > On 07/11/14 13:45, Radim Vansa wrote: >> Hijacking thread 'Remoting package refactor' as the discussion has shifted. >> >> Sure, AOP is another approach. However, besided another limitations, >> Byteman rules are quite fragile with respect to different versions: if >> you're injecting code based on internal implementation method, when the >> name/signature changes, the rule is broken. Sometimes you even have to >> use AT LINE to formulate the injection point. > Right. This is the same problem though as when support needs to create a > (e.f. one-off) patch to be applied by a customer: they need to grab the > exact same version the customer is running. > > So each diagnosis package would have to be dependent on the version (of > JGroups or JDG) used. Regardless of whether custom rules are added by a > support engineer, this has to be tested anyway before sending it off to > the customer. > >> Would you accept a compile-time dependency to some annotations package >> in JGroups that could 'tag' the injection points? The idea is that >> anyone changing the source code would move the injection point >> annotations as well. > You mean something like this ? > > @InjectionPoint("down") public void down(Event e) > > or > > @InjectingPoint ("num_msgs_sent") > protected int num_msgs_sent; > > No, this won't work... how would you do that ? Yes, this is the annotation syntax I had in mind, though, I was thinking about more high-level abstraction what's happening than just marking down injection points. Such as @ReceivedData public void receive(@From Address sender, byte[] data, int offset, @Size int length) {...} @ProcessingMessage protected void passMessageUp(@Message msg, ...) { ... } @ProcessingBatch protected void deliverBatch(@Batch MessageBatch batch) { ... } > > I don't really like this, on a general principle: AOP should *not* have > to change the src code in order to work. And the fact of the matter is > that you won't be able to identify *all* injection points beforehand... > unless you want to sprinkle your code with annotations. I have to agree with the fact that AOP should not have to change source. I had a special case in mind, that is tied to JGroups inspection and offers a way the monitoring with zero overhead when the monitoring is not in place. There, you'd just conceptually describe what JGroups does. > > >> I was already thinking about this in relation with Message Flow Tracer >> [1] (not working right now as the JGroups have changed since I was >> writing that)? > I took a quick look: nice ! > > This is exactly what I meant. Should be some sort of rule base in a VCS, > to which support engineers add rules when they have a case which > requires it and they deem it to be generally useful. > > Re API changes: doesn't Byteman have functionality which can check a > rule set against a code base (offline), to find out incompatibilities ? > Something like a static rule checker ? Right, this is possible - but you won't find if you've added another place that should be checked (e.g. MFT has to determine whether now you're processing a whole batch, or message alone - when you add a functionality to grab some stored messages and start processing them, as you do in UNICASTx, you won't spot that automatically). Beyond that, there are many false positives. E.g. if you have a never terminating loop in Runnable.run(), there is no place to inject the AT EXIT code and Byteman complains. In the end, human intervention is always required. Radim > >> Roman Macor is right now updating the rules and I was >> hoping that we could insert annotations into JGroups that would be used >> instead of the rules (I was already considering different AOP framework >> as Byteman does not allow AT EXIT to catch on leaving exceptions [2]). > Yes, I've also run into this before, not really nice. > >> Radim >> >> [1] https://github.com/rvansa/message-flow-tracer >> [2] https://issues.jboss.org/browse/BYTEMAN-237 >> >> On 11/07/2014 01:21 PM, Bela Ban wrote: >>> Hi Radim, >>> >>> no I haven't. However, you can replace the thread pools used by JGroups >>> and use custom pools. >>> >>> I like another idea better: inject Byteman code at runtime that keeps >>> track of this, and *other useful stats as well*. >>> >>> It would be very useful to support if we could ship a package to a >>> customer that is injected into their running system and grabs all the >>> vital stats we need for a few minutes, then removes itself again and >>> those stats are then sent to use as a ZIP file. >>> The good thing about byteman is that it can remove itself without a >>> trace; ie. there's no overhead before / after running byteman. >>> >>> >>> On 07/11/14 09:31, Radim Vansa wrote: >>>> Btw., have you ever considered checks if a thread returns to pool >>>> reasonably often? Some of the other datagrids use this, though there's >>>> not much how to react upon that beyond printing out stack traces (but >>>> you can at least report to management that some node seems to be broken). >>>> >>>> Radim >>>> >>>> On 11/07/2014 08:35 AM, Bela Ban wrote: >>>>> That's exactly what I suggested. No config gives you a shared global >>>>> thread pool for all caches. >>>>> >>>>> Those caches which need a separate pool can do that via configuration >>>>> (and of course also programmatically) >>>>> >>>>> On 06/11/14 20:31, Tristan Tarrant wrote: >>>>>> My opinion is that we should aim for less configuration, i.e. >>>>>> threadpools should mostly have sensible defaults and be shared by >>>>>> default unless there are extremely good reasons for not doing so. >>>>>> >>>>>> Tristan >>>>>> >>>>>> On 06/11/14 19:40, Radim Vansa wrote: >>>>>>> I second the opinion that any threadpools should be shared by default. >>>>>>> There are users who have hundreds or thousands of caches and having >>>>>>> separate threadpool for each of them could easily drain resources. And >>>>>>> sharing resources is the purpose of threadpools, right? >>>>>>> >>>>>>> Radim >>>>>>> >>>>>>> On 11/06/2014 04:37 PM, Bela Ban wrote: >>>>>>>> #1 I would by default have 1 thread pool shared by all caches >>>>>>>> #2 This global thread pool should be configurable, perhaps in the >>>>>>>> <global> section ? >>>>>>>> #3 Each cache by default uses the gobal thread pool >>>>>>>> #4 A cache can define its own thread pool, then it would use this one >>>>>>>> and not the global thread pool >>>>>>>> >>>>>>>> I think this gives you a mixture between ease of use and flexibility in >>>>>>>> configuring pool per cache if needed >>>>>>>> >>>>>>>> On 06/11/14 16:23, Pedro Ruivo wrote: >>>>>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>>>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>>>>>>>> * added a single thread remote executor service. This will handle the >>>>>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>>>>>>>> threads and with a new executor service, each cache can process their >>>>>>>>>>> own FIFO commands concurrently. >>>>>>>>>> +1000. This allows multiple updates from the same sender but to >>>>>>>>>> different caches to be executed in parallel, and will speed thing up. >>>>>>>>>> >>>>>>>>>> Do you intend to share a thread pool between the invocations handlers of >>>>>>>>>> the various caches, or do they each have their own thread pool ? Or is >>>>>>>>>> this configurable ? >>>>>>>>>> >>>>>>>>> That is question that cross my mind and I don't have any idea what would >>>>>>>>> be the best. So, for now, I will leave the thread pool shared between >>>>>>>>> the handlers. >>>>>>>>> >>>>>>>>> Never thought to make it configurable, but maybe that is the best >>>>>>>>> option. And maybe, it should be possible to have different max-thread >>>>>>>>> size per cache. For example: >>>>>>>>> >>>>>>>>> * all caches using this remote executor will share the same instance >>>>>>>>> <remote-executor name="shared" shared="true" max-threads=4.../> >>>>>>>>> >>>>>>>>> * all caches using this remote executor will create their own thread >>>>>>>>> pool with max-threads equals to 1 >>>>>>>>> <remote-executor name="low-throughput-cache" shared="false" >>>>>>>>> max-threads=1 .../> >>>>>>>>> >>>>>>>>> * all caches using this remote executor will create their own thread >>>>>>>>> pool with max-threads equals to 1000 >>>>>>>>> <remote executor name="high-throughput-cache" shared="false" >>>>>>>>> max-thread=1000 .../> >>>>>>>>> >>>>>>>>> is this what you have in mind? comments? >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Pedro >>>>>>>>> _______________________________________________ >>>>>>>>> infinispan-dev mailing list >>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>

No way I'd be aware of (you can specify the rule directly in annotation, but that's not what I'd like to do). Though, I don't think it would be too complicated to implement. But as I've said, I was inclining towards another AOP frameworks, or more low-level solutions such as Javassist.

For example similar tool Kamon [1] uses AspectJ Weaver. Roman, do you have the document describing pros and cons of those other AOP frameworks? [1] http://kamon.io/ On 11/10/2014 11:05 AM, Bela Ban wrote: > Does Byteman allow you to use annotations as injection points ? Didn't > know that. Can you show a sample RULE ? > > On 10/11/14 10:22, Radim Vansa wrote: >> On 11/07/2014 02:27 PM, Bela Ban wrote: >>> On 07/11/14 13:45, Radim Vansa wrote: >>>> Hijacking thread 'Remoting package refactor' as the discussion has shifted. >>>> >>>> Sure, AOP is another approach. However, besided another limitations, >>>> Byteman rules are quite fragile with respect to different versions: if >>>> you're injecting code based on internal implementation method, when the >>>> name/signature changes, the rule is broken. Sometimes you even have to >>>> use AT LINE to formulate the injection point. >>> Right. This is the same problem though as when support needs to create a >>> (e.f. one-off) patch to be applied by a customer: they need to grab the >>> exact same version the customer is running. >>> >>> So each diagnosis package would have to be dependent on the version (of >>> JGroups or JDG) used. Regardless of whether custom rules are added by a >>> support engineer, this has to be tested anyway before sending it off to >>> the customer. >>> >>>> Would you accept a compile-time dependency to some annotations package >>>> in JGroups that could 'tag' the injection points? The idea is that >>>> anyone changing the source code would move the injection point >>>> annotations as well. >>> You mean something like this ? >>> >>> @InjectionPoint("down") public void down(Event e) >>> >>> or >>> >>> @InjectingPoint ("num_msgs_sent") >>> protected int num_msgs_sent; >>> >>> No, this won't work... how would you do that ? >> Yes, this is the annotation syntax I had in mind, though, I was thinking >> about more high-level abstraction what's happening than just marking >> down injection points. >> Such as >> >> @ReceivedData >> public void receive(@From Address sender, byte[] data, int offset, @Size >> int length) {...} >> >> @ProcessingMessage >> protected void passMessageUp(@Message msg, ...) { ... } >> >> @ProcessingBatch >> protected void deliverBatch(@Batch MessageBatch batch) { ... } >> >> >>> I don't really like this, on a general principle: AOP should *not* have >>> to change the src code in order to work. And the fact of the matter is >>> that you won't be able to identify *all* injection points beforehand... >>> unless you want to sprinkle your code with annotations. >> I have to agree with the fact that AOP should not have to change source. >> I had a special case in mind, that is tied to JGroups inspection and >> offers a way the monitoring with zero overhead when the monitoring is >> not in place. There, you'd just conceptually describe what JGroups does. >> >>> >>>> I was already thinking about this in relation with Message Flow Tracer >>>> [1] (not working right now as the JGroups have changed since I was >>>> writing that)? >>> I took a quick look: nice ! >>> >>> This is exactly what I meant. Should be some sort of rule base in a VCS, >>> to which support engineers add rules when they have a case which >>> requires it and they deem it to be generally useful. >>> >>> Re API changes: doesn't Byteman have functionality which can check a >>> rule set against a code base (offline), to find out incompatibilities ? >>> Something like a static rule checker ? >> Right, this is possible - but you won't find if you've added another >> place that should be checked (e.g. MFT has to determine whether now >> you're processing a whole batch, or message alone - when you add a >> functionality to grab some stored messages and start processing them, as >> you do in UNICASTx, you won't spot that automatically). >> >> Beyond that, there are many false positives. E.g. if you have a never >> terminating loop in Runnable.run(), there is no place to inject the AT >> EXIT code and Byteman complains. >> >> In the end, human intervention is always required. >> >> Radim >> >>>> Roman Macor is right now updating the rules and I was >>>> hoping that we could insert annotations into JGroups that would be used >>>> instead of the rules (I was already considering different AOP framework >>>> as Byteman does not allow AT EXIT to catch on leaving exceptions [2]). >>> Yes, I've also run into this before, not really nice. >>> >>>> Radim >>>> >>>> [1] https://github.com/rvansa/message-flow-tracer >>>> [2] https://issues.jboss.org/browse/BYTEMAN-237 >>>> >>>> On 11/07/2014 01:21 PM, Bela Ban wrote: >>>>> Hi Radim, >>>>> >>>>> no I haven't. However, you can replace the thread pools used by JGroups >>>>> and use custom pools. >>>>> >>>>> I like another idea better: inject Byteman code at runtime that keeps >>>>> track of this, and *other useful stats as well*. >>>>> >>>>> It would be very useful to support if we could ship a package to a >>>>> customer that is injected into their running system and grabs all the >>>>> vital stats we need for a few minutes, then removes itself again and >>>>> those stats are then sent to use as a ZIP file. >>>>> The good thing about byteman is that it can remove itself without a >>>>> trace; ie. there's no overhead before / after running byteman. >>>>> >>>>> >>>>> On 07/11/14 09:31, Radim Vansa wrote: >>>>>> Btw., have you ever considered checks if a thread returns to pool >>>>>> reasonably often? Some of the other datagrids use this, though there's >>>>>> not much how to react upon that beyond printing out stack traces (but >>>>>> you can at least report to management that some node seems to be broken). >>>>>> >>>>>> Radim >>>>>> >>>>>> On 11/07/2014 08:35 AM, Bela Ban wrote: >>>>>>> That's exactly what I suggested. No config gives you a shared global >>>>>>> thread pool for all caches. >>>>>>> >>>>>>> Those caches which need a separate pool can do that via configuration >>>>>>> (and of course also programmatically) >>>>>>> >>>>>>> On 06/11/14 20:31, Tristan Tarrant wrote: >>>>>>>> My opinion is that we should aim for less configuration, i.e. >>>>>>>> threadpools should mostly have sensible defaults and be shared by >>>>>>>> default unless there are extremely good reasons for not doing so. >>>>>>>> >>>>>>>> Tristan >>>>>>>> >>>>>>>> On 06/11/14 19:40, Radim Vansa wrote: >>>>>>>>> I second the opinion that any threadpools should be shared by default. >>>>>>>>> There are users who have hundreds or thousands of caches and having >>>>>>>>> separate threadpool for each of them could easily drain resources. And >>>>>>>>> sharing resources is the purpose of threadpools, right? >>>>>>>>> >>>>>>>>> Radim >>>>>>>>> >>>>>>>>> On 11/06/2014 04:37 PM, Bela Ban wrote: >>>>>>>>>> #1 I would by default have 1 thread pool shared by all caches >>>>>>>>>> #2 This global thread pool should be configurable, perhaps in the >>>>>>>>>> <global> section ? >>>>>>>>>> #3 Each cache by default uses the gobal thread pool >>>>>>>>>> #4 A cache can define its own thread pool, then it would use this one >>>>>>>>>> and not the global thread pool >>>>>>>>>> >>>>>>>>>> I think this gives you a mixture between ease of use and flexibility in >>>>>>>>>> configuring pool per cache if needed >>>>>>>>>> >>>>>>>>>> On 06/11/14 16:23, Pedro Ruivo wrote: >>>>>>>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>>>>>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>>>>>>>>>> * added a single thread remote executor service. This will handle the >>>>>>>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>>>>>>>>>> threads and with a new executor service, each cache can process their >>>>>>>>>>>>> own FIFO commands concurrently. >>>>>>>>>>>> +1000. This allows multiple updates from the same sender but to >>>>>>>>>>>> different caches to be executed in parallel, and will speed thing up. >>>>>>>>>>>> >>>>>>>>>>>> Do you intend to share a thread pool between the invocations handlers of >>>>>>>>>>>> the various caches, or do they each have their own thread pool ? Or is >>>>>>>>>>>> this configurable ? >>>>>>>>>>>> >>>>>>>>>>> That is question that cross my mind and I don't have any idea what would >>>>>>>>>>> be the best. So, for now, I will leave the thread pool shared between >>>>>>>>>>> the handlers. >>>>>>>>>>> >>>>>>>>>>> Never thought to make it configurable, but maybe that is the best >>>>>>>>>>> option. And maybe, it should be possible to have different max-thread >>>>>>>>>>> size per cache. For example: >>>>>>>>>>> >>>>>>>>>>> * all caches using this remote executor will share the same instance >>>>>>>>>>> <remote-executor name="shared" shared="true" max-threads=4.../> >>>>>>>>>>> >>>>>>>>>>> * all caches using this remote executor will create their own thread >>>>>>>>>>> pool with max-threads equals to 1 >>>>>>>>>>> <remote-executor name="low-throughput-cache" shared="false" >>>>>>>>>>> max-threads=1 .../> >>>>>>>>>>> >>>>>>>>>>> * all caches using this remote executor will create their own thread >>>>>>>>>>> pool with max-threads equals to 1000 >>>>>>>>>>> <remote executor name="high-throughput-cache" shared="false" >>>>>>>>>>> max-thread=1000 .../> >>>>>>>>>>> >>>>>>>>>>> is this what you have in mind? comments? >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Pedro >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> infinispan-dev mailing list >>>>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> infinispan-dev mailing list >>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>> >>

Could we just have a listener interface rather than playing with AOP? You don't want to support those libraries ;-) For example Hibernate is deprecating javassist. Sanne On 10 November 2014 10:41, Bela Ban <bban(a)redhat.com&gt; wrote: > > > On 10/11/14 11:33, Radim Vansa wrote: >> No way I'd be aware of (you can specify the rule directly in annotation, >> but that's not what I'd like to do). Though, I don't think it would be >> too complicated to implement. >> But as I've said, I was inclining towards another AOP frameworks, or >> more low-level solutions such as Javassist. > > What's the benefit of this ? I don't think you could define the > joinpoint in a strongly-typed fashion, so refactoring would not work > either if you for example change a method name. Or would it ? > >> For example similar tool >> Kamon [1] uses AspectJ Weaver. >> >> Roman, do you have the document describing pros and cons of those other >> AOP frameworks? >> >> [1] http://kamon.io/ >> >> On 11/10/2014 11:05 AM, Bela Ban wrote: >>> Does Byteman allow you to use annotations as injection points ? Didn't >>> know that. Can you show a sample RULE ? >>> >>> On 10/11/14 10:22, Radim Vansa wrote: >>>> On 11/07/2014 02:27 PM, Bela Ban wrote: >>>>> On 07/11/14 13:45, Radim Vansa wrote: >>>>>> Hijacking thread 'Remoting package refactor' as the discussion has shifted. >>>>>> >>>>>> Sure, AOP is another approach. However, besided another limitations, >>>>>> Byteman rules are quite fragile with respect to different versions: if >>>>>> you're injecting code based on internal implementation method, when the >>>>>> name/signature changes, the rule is broken. Sometimes you even have to >>>>>> use AT LINE to formulate the injection point. >>>>> Right. This is the same problem though as when support needs to create a >>>>> (e.f. one-off) patch to be applied by a customer: they need to grab the >>>>> exact same version the customer is running. >>>>> >>>>> So each diagnosis package would have to be dependent on the version (of >>>>> JGroups or JDG) used. Regardless of whether custom rules are added by a >>>>> support engineer, this has to be tested anyway before sending it off to >>>>> the customer. >>>>> >>>>>> Would you accept a compile-time dependency to some annotations package >>>>>> in JGroups that could 'tag' the injection points? The idea is that >>>>>> anyone changing the source code would move the injection point >>>>>> annotations as well. >>>>> You mean something like this ? >>>>> >>>>> @InjectionPoint("down") public void down(Event e) >>>>> >>>>> or >>>>> >>>>> @InjectingPoint ("num_msgs_sent") >>>>> protected int num_msgs_sent; >>>>> >>>>> No, this won't work... how would you do that ? >>>> Yes, this is the annotation syntax I had in mind, though, I was thinking >>>> about more high-level abstraction what's happening than just marking >>>> down injection points. >>>> Such as >>>> >>>> @ReceivedData >>>> public void receive(@From Address sender, byte[] data, int offset, @Size >>>> int length) {...} >>>> >>>> @ProcessingMessage >>>> protected void passMessageUp(@Message msg, ...) { ... } >>>> >>>> @ProcessingBatch >>>> protected void deliverBatch(@Batch MessageBatch batch) { ... } >>>> >>>> >>>>> I don't really like this, on a general principle: AOP should *not* have >>>>> to change the src code in order to work. And the fact of the matter is >>>>> that you won't be able to identify *all* injection points beforehand... >>>>> unless you want to sprinkle your code with annotations. >>>> I have to agree with the fact that AOP should not have to change source. >>>> I had a special case in mind, that is tied to JGroups inspection and >>>> offers a way the monitoring with zero overhead when the monitoring is >>>> not in place. There, you'd just conceptually describe what JGroups does. >>>> >>>>> >>>>>> I was already thinking about this in relation with Message Flow Tracer >>>>>> [1] (not working right now as the JGroups have changed since I was >>>>>> writing that)? >>>>> I took a quick look: nice ! >>>>> >>>>> This is exactly what I meant. Should be some sort of rule base in a VCS, >>>>> to which support engineers add rules when they have a case which >>>>> requires it and they deem it to be generally useful. >>>>> >>>>> Re API changes: doesn't Byteman have functionality which can check a >>>>> rule set against a code base (offline), to find out incompatibilities ? >>>>> Something like a static rule checker ? >>>> Right, this is possible - but you won't find if you've added another >>>> place that should be checked (e.g. MFT has to determine whether now >>>> you're processing a whole batch, or message alone - when you add a >>>> functionality to grab some stored messages and start processing them, as >>>> you do in UNICASTx, you won't spot that automatically). >>>> >>>> Beyond that, there are many false positives. E.g. if you have a never >>>> terminating loop in Runnable.run(), there is no place to inject the AT >>>> EXIT code and Byteman complains. >>>> >>>> In the end, human intervention is always required. >>>> >>>> Radim >>>> >>>>>> Roman Macor is right now updating the rules and I was >>>>>> hoping that we could insert annotations into JGroups that would be used >>>>>> instead of the rules (I was already considering different AOP framework >>>>>> as Byteman does not allow AT EXIT to catch on leaving exceptions [2]). >>>>> Yes, I've also run into this before, not really nice. >>>>> >>>>>> Radim >>>>>> >>>>>> [1] https://github.com/rvansa/message-flow-tracer >>>>>> [2] https://issues.jboss.org/browse/BYTEMAN-237 >>>>>> >>>>>> On 11/07/2014 01:21 PM, Bela Ban wrote: >>>>>>> Hi Radim, >>>>>>> >>>>>>> no I haven't. However, you can replace the thread pools used by JGroups >>>>>>> and use custom pools. >>>>>>> >>>>>>> I like another idea better: inject Byteman code at runtime that keeps >>>>>>> track of this, and *other useful stats as well*. >>>>>>> >>>>>>> It would be very useful to support if we could ship a package to a >>>>>>> customer that is injected into their running system and grabs all the >>>>>>> vital stats we need for a few minutes, then removes itself again and >>>>>>> those stats are then sent to use as a ZIP file. >>>>>>> The good thing about byteman is that it can remove itself without a >>>>>>> trace; ie. there's no overhead before / after running byteman. >>>>>>> >>>>>>> >>>>>>> On 07/11/14 09:31, Radim Vansa wrote: >>>>>>>> Btw., have you ever considered checks if a thread returns to pool >>>>>>>> reasonably often? Some of the other datagrids use this, though there's >>>>>>>> not much how to react upon that beyond printing out stack traces (but >>>>>>>> you can at least report to management that some node seems to be broken). >>>>>>>> >>>>>>>> Radim >>>>>>>> >>>>>>>> On 11/07/2014 08:35 AM, Bela Ban wrote: >>>>>>>>> That's exactly what I suggested. No config gives you a shared global >>>>>>>>> thread pool for all caches. >>>>>>>>> >>>>>>>>> Those caches which need a separate pool can do that via configuration >>>>>>>>> (and of course also programmatically) >>>>>>>>> >>>>>>>>> On 06/11/14 20:31, Tristan Tarrant wrote: >>>>>>>>>> My opinion is that we should aim for less configuration, i.e. >>>>>>>>>> threadpools should mostly have sensible defaults and be shared by >>>>>>>>>> default unless there are extremely good reasons for not doing so. >>>>>>>>>> >>>>>>>>>> Tristan >>>>>>>>>> >>>>>>>>>> On 06/11/14 19:40, Radim Vansa wrote: >>>>>>>>>>> I second the opinion that any threadpools should be shared by default. >>>>>>>>>>> There are users who have hundreds or thousands of caches and having >>>>>>>>>>> separate threadpool for each of them could easily drain resources. And >>>>>>>>>>> sharing resources is the purpose of threadpools, right? >>>>>>>>>>> >>>>>>>>>>> Radim >>>>>>>>>>> >>>>>>>>>>> On 11/06/2014 04:37 PM, Bela Ban wrote: >>>>>>>>>>>> #1 I would by default have 1 thread pool shared by all caches >>>>>>>>>>>> #2 This global thread pool should be configurable, perhaps in the >>>>>>>>>>>> <global> section ? >>>>>>>>>>>> #3 Each cache by default uses the gobal thread pool >>>>>>>>>>>> #4 A cache can define its own thread pool, then it would use this one >>>>>>>>>>>> and not the global thread pool >>>>>>>>>>>> >>>>>>>>>>>> I think this gives you a mixture between ease of use and flexibility in >>>>>>>>>>>> configuring pool per cache if needed >>>>>>>>>>>> >>>>>>>>>>>> On 06/11/14 16:23, Pedro Ruivo wrote: >>>>>>>>>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>>>>>>>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>>>>>>>>>>>> * added a single thread remote executor service. This will handle the >>>>>>>>>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>>>>>>>>>>>> threads and with a new executor service, each cache can process their >>>>>>>>>>>>>>> own FIFO commands concurrently. >>>>>>>>>>>>>> +1000. This allows multiple updates from the same sender but to >>>>>>>>>>>>>> different caches to be executed in parallel, and will speed thing up. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Do you intend to share a thread pool between the invocations handlers of >>>>>>>>>>>>>> the various caches, or do they each have their own thread pool ? Or is >>>>>>>>>>>>>> this configurable ? >>>>>>>>>>>>>> >>>>>>>>>>>>> That is question that cross my mind and I don't have any idea what would >>>>>>>>>>>>> be the best. So, for now, I will leave the thread pool shared between >>>>>>>>>>>>> the handlers. >>>>>>>>>>>>> >>>>>>>>>>>>> Never thought to make it configurable, but maybe that is the best >>>>>>>>>>>>> option. And maybe, it should be possible to have different max-thread >>>>>>>>>>>>> size per cache. For example: >>>>>>>>>>>>> >>>>>>>>>>>>> * all caches using this remote executor will share the same instance >>>>>>>>>>>>> <remote-executor name="shared" shared="true" max-threads=4.../> >>>>>>>>>>>>> >>>>>>>>>>>>> * all caches using this remote executor will create their own thread >>>>>>>>>>>>> pool with max-threads equals to 1 >>>>>>>>>>>>> <remote-executor name="low-throughput-cache" shared="false" >>>>>>>>>>>>> max-threads=1 .../> >>>>>>>>>>>>> >>>>>>>>>>>>> * all caches using this remote executor will create their own thread >>>>>>>>>>>>> pool with max-threads equals to 1000 >>>>>>>>>>>>> <remote executor name="high-throughput-cache" shared="false" >>>>>>>>>>>>> max-thread=1000 .../> >>>>>>>>>>>>> >>>>>>>>>>>>> is this what you have in mind? comments? >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Pedro >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> infinispan-dev mailing list >>>>>>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> infinispan-dev mailing list >>>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>> >>>> >> >> > > -- > Bela Ban, JGroups lead (http://www.jgroups.org) > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10/11/14 11:33, Radim Vansa wrote: > No way I'd be aware of (you can specify the rule directly in annotation, > but that's not what I'd like to do). Though, I don't think it would be > too complicated to implement. > But as I've said, I was inclining towards another AOP frameworks, or > more low-level solutions such as Javassist. What's the benefit of this ? I don't think you could define the joinpoint in a strongly-typed fashion, so refactoring would not work either if you for example change a method name. Or would it ?

> For example similar tool > Kamon [1] uses AspectJ Weaver. > > Roman, do you have the document describing pros and cons of those other > AOP frameworks? > > [1] http://kamon.io/ > > On 11/10/2014 11:05 AM, Bela Ban wrote: >> Does Byteman allow you to use annotations as injection points ? Didn't >> know that. Can you show a sample RULE ? >> >> On 10/11/14 10:22, Radim Vansa wrote: >>> On 11/07/2014 02:27 PM, Bela Ban wrote: >>>> On 07/11/14 13:45, Radim Vansa wrote: >>>>> Hijacking thread 'Remoting package refactor' as the discussion has shifted. >>>>> >>>>> Sure, AOP is another approach. However, besided another limitations, >>>>> Byteman rules are quite fragile with respect to different versions: if >>>>> you're injecting code based on internal implementation method, when the >>>>> name/signature changes, the rule is broken. Sometimes you even have to >>>>> use AT LINE to formulate the injection point. >>>> Right. This is the same problem though as when support needs to create a >>>> (e.f. one-off) patch to be applied by a customer: they need to grab the >>>> exact same version the customer is running. >>>> >>>> So each diagnosis package would have to be dependent on the version (of >>>> JGroups or JDG) used. Regardless of whether custom rules are added by a >>>> support engineer, this has to be tested anyway before sending it off to >>>> the customer. >>>> >>>>> Would you accept a compile-time dependency to some annotations package >>>>> in JGroups that could 'tag' the injection points? The idea is that >>>>> anyone changing the source code would move the injection point >>>>> annotations as well. >>>> You mean something like this ? >>>> >>>> @InjectionPoint("down") public void down(Event e) >>>> >>>> or >>>> >>>> @InjectingPoint ("num_msgs_sent") >>>> protected int num_msgs_sent; >>>> >>>> No, this won't work... how would you do that ? >>> Yes, this is the annotation syntax I had in mind, though, I was thinking >>> about more high-level abstraction what's happening than just marking >>> down injection points. >>> Such as >>> >>> @ReceivedData >>> public void receive(@From Address sender, byte[] data, int offset, @Size >>> int length) {...} >>> >>> @ProcessingMessage >>> protected void passMessageUp(@Message msg, ...) { ... } >>> >>> @ProcessingBatch >>> protected void deliverBatch(@Batch MessageBatch batch) { ... } >>> >>> >>>> I don't really like this, on a general principle: AOP should *not* have >>>> to change the src code in order to work. And the fact of the matter is >>>> that you won't be able to identify *all* injection points beforehand... >>>> unless you want to sprinkle your code with annotations. >>> I have to agree with the fact that AOP should not have to change source. >>> I had a special case in mind, that is tied to JGroups inspection and >>> offers a way the monitoring with zero overhead when the monitoring is >>> not in place. There, you'd just conceptually describe what JGroups does. >>> >>>>> I was already thinking about this in relation with Message Flow Tracer >>>>> [1] (not working right now as the JGroups have changed since I was >>>>> writing that)? >>>> I took a quick look: nice ! >>>> >>>> This is exactly what I meant. Should be some sort of rule base in a VCS, >>>> to which support engineers add rules when they have a case which >>>> requires it and they deem it to be generally useful. >>>> >>>> Re API changes: doesn't Byteman have functionality which can check a >>>> rule set against a code base (offline), to find out incompatibilities ? >>>> Something like a static rule checker ? >>> Right, this is possible - but you won't find if you've added another >>> place that should be checked (e.g. MFT has to determine whether now >>> you're processing a whole batch, or message alone - when you add a >>> functionality to grab some stored messages and start processing them, as >>> you do in UNICASTx, you won't spot that automatically). >>> >>> Beyond that, there are many false positives. E.g. if you have a never >>> terminating loop in Runnable.run(), there is no place to inject the AT >>> EXIT code and Byteman complains. >>> >>> In the end, human intervention is always required. >>> >>> Radim >>> >>>>> Roman Macor is right now updating the rules and I was >>>>> hoping that we could insert annotations into JGroups that would be used >>>>> instead of the rules (I was already considering different AOP framework >>>>> as Byteman does not allow AT EXIT to catch on leaving exceptions [2]). >>>> Yes, I've also run into this before, not really nice. >>>> >>>>> Radim >>>>> >>>>> [1] https://github.com/rvansa/message-flow-tracer >>>>> [2] https://issues.jboss.org/browse/BYTEMAN-237 >>>>> >>>>> On 11/07/2014 01:21 PM, Bela Ban wrote: >>>>>> Hi Radim, >>>>>> >>>>>> no I haven't. However, you can replace the thread pools used by JGroups >>>>>> and use custom pools. >>>>>> >>>>>> I like another idea better: inject Byteman code at runtime that keeps >>>>>> track of this, and *other useful stats as well*. >>>>>> >>>>>> It would be very useful to support if we could ship a package to a >>>>>> customer that is injected into their running system and grabs all the >>>>>> vital stats we need for a few minutes, then removes itself again and >>>>>> those stats are then sent to use as a ZIP file. >>>>>> The good thing about byteman is that it can remove itself without a >>>>>> trace; ie. there's no overhead before / after running byteman. >>>>>> >>>>>> >>>>>> On 07/11/14 09:31, Radim Vansa wrote: >>>>>>> Btw., have you ever considered checks if a thread returns to pool >>>>>>> reasonably often? Some of the other datagrids use this, though there's >>>>>>> not much how to react upon that beyond printing out stack traces (but >>>>>>> you can at least report to management that some node seems to be broken). >>>>>>> >>>>>>> Radim >>>>>>> >>>>>>> On 11/07/2014 08:35 AM, Bela Ban wrote: >>>>>>>> That's exactly what I suggested. No config gives you a shared global >>>>>>>> thread pool for all caches. >>>>>>>> >>>>>>>> Those caches which need a separate pool can do that via configuration >>>>>>>> (and of course also programmatically) >>>>>>>> >>>>>>>> On 06/11/14 20:31, Tristan Tarrant wrote: >>>>>>>>> My opinion is that we should aim for less configuration, i.e. >>>>>>>>> threadpools should mostly have sensible defaults and be shared by >>>>>>>>> default unless there are extremely good reasons for not doing so. >>>>>>>>> >>>>>>>>> Tristan >>>>>>>>> >>>>>>>>> On 06/11/14 19:40, Radim Vansa wrote: >>>>>>>>>> I second the opinion that any threadpools should be shared by default. >>>>>>>>>> There are users who have hundreds or thousands of caches and having >>>>>>>>>> separate threadpool for each of them could easily drain resources. And >>>>>>>>>> sharing resources is the purpose of threadpools, right? >>>>>>>>>> >>>>>>>>>> Radim >>>>>>>>>> >>>>>>>>>> On 11/06/2014 04:37 PM, Bela Ban wrote: >>>>>>>>>>> #1 I would by default have 1 thread pool shared by all caches >>>>>>>>>>> #2 This global thread pool should be configurable, perhaps in the >>>>>>>>>>> <global> section ? >>>>>>>>>>> #3 Each cache by default uses the gobal thread pool >>>>>>>>>>> #4 A cache can define its own thread pool, then it would use this one >>>>>>>>>>> and not the global thread pool >>>>>>>>>>> >>>>>>>>>>> I think this gives you a mixture between ease of use and flexibility in >>>>>>>>>>> configuring pool per cache if needed >>>>>>>>>>> >>>>>>>>>>> On 06/11/14 16:23, Pedro Ruivo wrote: >>>>>>>>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>>>>>>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>>>>>>>>>>> * added a single thread remote executor service. This will handle the >>>>>>>>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>>>>>>>>>>> threads and with a new executor service, each cache can process their >>>>>>>>>>>>>> own FIFO commands concurrently. >>>>>>>>>>>>> +1000. This allows multiple updates from the same sender but to >>>>>>>>>>>>> different caches to be executed in parallel, and will speed thing up. >>>>>>>>>>>>> >>>>>>>>>>>>> Do you intend to share a thread pool between the invocations handlers of >>>>>>>>>>>>> the various caches, or do they each have their own thread pool ? Or is >>>>>>>>>>>>> this configurable ? >>>>>>>>>>>>> >>>>>>>>>>>> That is question that cross my mind and I don't have any idea what would >>>>>>>>>>>> be the best. So, for now, I will leave the thread pool shared between >>>>>>>>>>>> the handlers. >>>>>>>>>>>> >>>>>>>>>>>> Never thought to make it configurable, but maybe that is the best >>>>>>>>>>>> option. And maybe, it should be possible to have different max-thread >>>>>>>>>>>> size per cache. For example: >>>>>>>>>>>> >>>>>>>>>>>> * all caches using this remote executor will share the same instance >>>>>>>>>>>> <remote-executor name="shared" shared="true" max-threads=4.../> >>>>>>>>>>>> >>>>>>>>>>>> * all caches using this remote executor will create their own thread >>>>>>>>>>>> pool with max-threads equals to 1 >>>>>>>>>>>> <remote-executor name="low-throughput-cache" shared="false" >>>>>>>>>>>> max-threads=1 .../> >>>>>>>>>>>> >>>>>>>>>>>> * all caches using this remote executor will create their own thread >>>>>>>>>>>> pool with max-threads equals to 1000 >>>>>>>>>>>> <remote executor name="high-throughput-cache" shared="false" >>>>>>>>>>>> max-thread=1000 .../> >>>>>>>>>>>> >>>>>>>>>>>> is this what you have in mind? comments? >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Pedro >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> infinispan-dev mailing list >>>>>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> infinispan-dev mailing list >>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>> >

@Pedro, did you consider using a ForkJoinPool instead? Traditional JDK pools are known to be very hard to configure and get it “right”. Fork join pools are being used as default thread pools in other libraries, vastly reducing configuration. Jessitron has published some interesting blog posts on the advantages of traditional ExecutorService vs Fork/Join pools and viceversa. See [1] and [3]. She also did a talk on it, see [4]. Cheers, p.s. I’ve not studied your use case in depth to decide whether F/J would suite better, but it’s certainly worth a look now that we’re on Java 7. [1] https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinPo... [2] http://blog.jessitron.com/2014/01/choosing-executorservice.html [3] http://blog.jessitron.com/2014/02/scala-global-executioncontext-makes.html [4] https://www.youtube.com/watch?v=yhguOt863nw On 07 Nov 2014, at 09:31, Radim Vansa <rvansa(a)redhat.com&gt; wrote: > Btw., have you ever considered checks if a thread returns to pool > reasonably often? Some of the other datagrids use this, though there's > not much how to react upon that beyond printing out stack traces (but > you can at least report to management that some node seems to be broken). > > Radim > > On 11/07/2014 08:35 AM, Bela Ban wrote: >> That's exactly what I suggested. No config gives you a shared global >> thread pool for all caches. >> >> Those caches which need a separate pool can do that via configuration >> (and of course also programmatically) >> >> On 06/11/14 20:31, Tristan Tarrant wrote: >>> My opinion is that we should aim for less configuration, i.e. >>> threadpools should mostly have sensible defaults and be shared by >>> default unless there are extremely good reasons for not doing so. >>> >>> Tristan >>> >>> On 06/11/14 19:40, Radim Vansa wrote: >>>> I second the opinion that any threadpools should be shared by default. >>>> There are users who have hundreds or thousands of caches and having >>>> separate threadpool for each of them could easily drain resources. And >>>> sharing resources is the purpose of threadpools, right? >>>> >>>> Radim >>>> >>>> On 11/06/2014 04:37 PM, Bela Ban wrote: >>>>> #1 I would by default have 1 thread pool shared by all caches >>>>> #2 This global thread pool should be configurable, perhaps in the >>>>> <global> section ? >>>>> #3 Each cache by default uses the gobal thread pool >>>>> #4 A cache can define its own thread pool, then it would use this one >>>>> and not the global thread pool >>>>> >>>>> I think this gives you a mixture between ease of use and flexibility in >>>>> configuring pool per cache if needed >>>>> >>>>> On 06/11/14 16:23, Pedro Ruivo wrote: >>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: >>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: >>>>>>>> * added a single thread remote executor service. This will handle the >>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming >>>>>>>> threads and with a new executor service, each cache can process their >>>>>>>> own FIFO commands concurrently. >>>>>>> +1000. This allows multiple updates from the same sender but to >>>>>>> different caches to be executed in parallel, and will speed thing up. >>>>>>> >>>>>>> Do you intend to share a thread pool between the invocations handlers of >>>>>>> the various caches, or do they each have their own thread pool ? Or is >>>>>>> this configurable ? >>>>>>> >>>>>> That is question that cross my mind and I don't have any idea what would >>>>>> be the best. So, for now, I will leave the thread pool shared between >>>>>> the handlers. >>>>>> >>>>>> Never thought to make it configurable, but maybe that is the best >>>>>> option. And maybe, it should be possible to have different max-thread >>>>>> size per cache. For example: >>>>>> >>>>>> * all caches using this remote executor will share the same instance >>>>>> <remote-executor name="shared" shared="true" max-threads=4.../> >>>>>> >>>>>> * all caches using this remote executor will create their own thread >>>>>> pool with max-threads equals to 1 >>>>>> <remote-executor name="low-throughput-cache" shared="false" >>>>>> max-threads=1 .../> >>>>>> >>>>>> * all caches using this remote executor will create their own thread >>>>>> pool with max-threads equals to 1000 >>>>>> <remote executor name="high-throughput-cache" shared="false" >>>>>> max-thread=1000 .../> >>>>>> >>>>>> is this what you have in mind? comments? >>>>>> >>>>>> Cheers, >>>>>> Pedro >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> > > -- > Radim Vansa <rvansa(a)redhat.com&gt; > JBoss DataGrid QA > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Radim, I also knew the 1.7 ForkJoinPool isn't really optimized for blocking tasks, but the ManagedBlocker interface mentioned in [3] seems to be intended just for that.

Re: commonPool(), we can (and should) still create our own ForkJoinPool instead of using the global one.

Cheers Dan On Thu, Nov 13, 2014 at 9:57 AM, Radim Vansa <rvansa(a)redhat.com&gt; wrote: > > F/J tasks should not acquire any locks (or, generally, block) during > their execution. At least according to JavaDocs. Are we ready for that? > > Btw., I really don't like the fact that the commonPool() cannot be > properly shutdown. This leads to threadlocal variables leaking when the > component using F/J pool is undeployed (the classloader cannot be GCed > and you end up with OOME in PermGen space). > > Radim > > On 11/13/2014 08:28 AM, Galder Zamarreño wrote: > > @Pedro, did you consider using a ForkJoinPool instead? > > > > Traditional JDK pools are known to be very hard to configure and get it > > “right”. Fork join pools are being used as default thread pools in other > > libraries, vastly reducing configuration. > > > > Jessitron has published some interesting blog posts on the advantages of > > traditional ExecutorService vs Fork/Join pools and viceversa. See [1] and > > [3]. She also did a talk on it, see [4]. > > > > Cheers, > > > > p.s. I’ve not studied your use case in depth to decide whether F/J would > > suite better, but it’s certainly worth a look now that we’re on Java 7. > > > > [1] > > https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinPo... > > [2] http://blog.jessitron.com/2014/01/choosing-executorservice.html > > [3] > > http://blog.jessitron.com/2014/02/scala-global-executioncontext-makes.html > > [4] https://www.youtube.com/watch?v=yhguOt863nw > > > > On 07 Nov 2014, at 09:31, Radim Vansa <rvansa(a)redhat.com&gt; wrote: > > > >> Btw., have you ever considered checks if a thread returns to pool > >> reasonably often? Some of the other datagrids use this, though there's > >> not much how to react upon that beyond printing out stack traces (but > >> you can at least report to management that some node seems to be > >> broken). > >> > >> Radim > >> > >> On 11/07/2014 08:35 AM, Bela Ban wrote: > >>> That's exactly what I suggested. No config gives you a shared global > >>> thread pool for all caches. > >>> > >>> Those caches which need a separate pool can do that via configuration > >>> (and of course also programmatically) > >>> > >>> On 06/11/14 20:31, Tristan Tarrant wrote: > >>>> My opinion is that we should aim for less configuration, i.e. > >>>> threadpools should mostly have sensible defaults and be shared by > >>>> default unless there are extremely good reasons for not doing so. > >>>> > >>>> Tristan > >>>> > >>>> On 06/11/14 19:40, Radim Vansa wrote: > >>>>> I second the opinion that any threadpools should be shared by > >>>>> default. > >>>>> There are users who have hundreds or thousands of caches and having > >>>>> separate threadpool for each of them could easily drain resources. > >>>>> And > >>>>> sharing resources is the purpose of threadpools, right? > >>>>> > >>>>> Radim > >>>>> > >>>>> On 11/06/2014 04:37 PM, Bela Ban wrote: > >>>>>> #1 I would by default have 1 thread pool shared by all caches > >>>>>> #2 This global thread pool should be configurable, perhaps in the > >>>>>> <global> section ? > >>>>>> #3 Each cache by default uses the gobal thread pool > >>>>>> #4 A cache can define its own thread pool, then it would use this > >>>>>> one > >>>>>> and not the global thread pool > >>>>>> > >>>>>> I think this gives you a mixture between ease of use and > >>>>>> flexibility in > >>>>>> configuring pool per cache if needed > >>>>>> > >>>>>> On 06/11/14 16:23, Pedro Ruivo wrote: > >>>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote: > >>>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote: > >>>>>>>>> * added a single thread remote executor service. This will > >>>>>>>>> handle the > >>>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups > >>>>>>>>> incoming > >>>>>>>>> threads and with a new executor service, each cache can process > >>>>>>>>> their > >>>>>>>>> own FIFO commands concurrently. > >>>>>>>> +1000. This allows multiple updates from the same sender but to > >>>>>>>> different caches to be executed in parallel, and will speed thing > >>>>>>>> up. > >>>>>>>> > >>>>>>>> Do you intend to share a thread pool between the invocations > >>>>>>>> handlers of > >>>>>>>> the various caches, or do they each have their own thread pool ? > >>>>>>>> Or is > >>>>>>>> this configurable ? > >>>>>>>> > >>>>>>> That is question that cross my mind and I don't have any idea what > >>>>>>> would > >>>>>>> be the best. So, for now, I will leave the thread pool shared > >>>>>>> between > >>>>>>> the handlers. > >>>>>>> > >>>>>>> Never thought to make it configurable, but maybe that is the best > >>>>>>> option. And maybe, it should be possible to have different > >>>>>>> max-thread > >>>>>>> size per cache. For example: > >>>>>>> > >>>>>>> * all caches using this remote executor will share the same > >>>>>>> instance > >>>>>>> <remote-executor name="shared" shared="true" max-threads=4.../> > >>>>>>> > >>>>>>> * all caches using this remote executor will create their own > >>>>>>> thread > >>>>>>> pool with max-threads equals to 1 > >>>>>>> <remote-executor name="low-throughput-cache" shared="false" > >>>>>>> max-threads=1 .../> > >>>>>>> > >>>>>>> * all caches using this remote executor will create their own > >>>>>>> thread > >>>>>>> pool with max-threads equals to 1000 > >>>>>>> <remote executor name="high-throughput-cache" shared="false" > >>>>>>> max-thread=1000 .../> > >>>>>>> > >>>>>>> is this what you have in mind? comments? > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Pedro > >>>>>>> _______________________________________________ > >>>>>>> infinispan-dev mailing list > >>>>>>> infinispan-dev(a)lists.jboss.org > >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev > >>>>>>> > >>>> _______________________________________________ > >>>> infinispan-dev mailing list > >>>> infinispan-dev(a)lists.jboss.org > >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev > >>>> > >> > >> -- > >> Radim Vansa <rvansa(a)redhat.com&gt; > >> JBoss DataGrid QA > >> > >> _______________________________________________ > >> infinispan-dev mailing list > >> infinispan-dev(a)lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > -- > > Galder Zamarreño > > galder(a)redhat.com > > twitter.com/galderz > > > > > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Radim Vansa <rvansa(a)redhat.com&gt; > JBoss DataGrid QA > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev