[infinispan-dev] Thread pools monitoring

Bela Ban bban at redhat.com
Mon Nov 10 06:08:37 EST 2014 

The drawbacks of a listener interface are:
- You don't know up-front which methods or attributes you want to listen on
- You need code in your application to provide listener registration and 
instrument your code to notify listeners

These add code to the app which has nothing to do with operational 
semantics and IMO eliminating this code is exactly one of the strong 
points of AOP.

On 10/11/14 11:49, Sanne Grinovero wrote:
> Could we just have a listener interface rather than playing with AOP?
>
> You don't want to support those libraries ;-) For example Hibernate is
> deprecating javassist.
>
> Sanne
>
> On 10 November 2014 10:41, Bela Ban <bban at redhat.com> wrote:
>>
>>
>> On 10/11/14 11:33, Radim Vansa wrote:
>>> No way I'd be aware of (you can specify the rule directly in annotation,
>>> but that's not what I'd like to do). Though, I don't think it would be
>>> too complicated to implement.
>>> But as I've said, I was inclining towards another AOP frameworks, or
>>> more low-level solutions such as Javassist.
>>
>> What's the benefit of this ? I don't think you could define the
>> joinpoint in a strongly-typed fashion, so refactoring would not work
>> either if you for example change a method name. Or would it ?
>>
>>> For example similar tool
>>> Kamon [1] uses AspectJ Weaver.
>>>
>>> Roman, do you have the document describing pros and cons of those other
>>> AOP frameworks?
>>>
>>> [1] http://kamon.io/
>>>
>>> On 11/10/2014 11:05 AM, Bela Ban wrote:
>>>> Does Byteman allow you to use annotations as injection points ? Didn't
>>>> know that. Can you show a sample RULE ?
>>>>
>>>> On 10/11/14 10:22, Radim Vansa wrote:
>>>>> On 11/07/2014 02:27 PM, Bela Ban wrote:
>>>>>> On 07/11/14 13:45, Radim Vansa wrote:
>>>>>>> Hijacking thread 'Remoting package refactor' as the discussion has shifted.
>>>>>>>
>>>>>>> Sure, AOP is another approach. However, besided another limitations,
>>>>>>> Byteman rules are quite fragile with respect to different versions: if
>>>>>>> you're injecting code based on internal implementation method, when the
>>>>>>> name/signature changes, the rule is broken. Sometimes you even have to
>>>>>>> use AT LINE to formulate the injection point.
>>>>>> Right. This is the same problem though as when support needs to create a
>>>>>> (e.f. one-off) patch to be applied by a customer: they need to grab the
>>>>>> exact same version the customer is running.
>>>>>>
>>>>>> So each diagnosis package would have to be dependent on the version (of
>>>>>> JGroups or JDG) used. Regardless of whether custom rules are added by a
>>>>>> support engineer, this has to be tested anyway before sending it off to
>>>>>> the customer.
>>>>>>
>>>>>>> Would you accept a compile-time dependency to some annotations package
>>>>>>> in JGroups that could 'tag' the injection points? The idea is that
>>>>>>> anyone changing the source code would move the injection point
>>>>>>> annotations as well.
>>>>>> You mean something like this ?
>>>>>>
>>>>>> @InjectionPoint("down") public void down(Event e)
>>>>>>
>>>>>> or
>>>>>>
>>>>>> @InjectingPoint ("num_msgs_sent")
>>>>>> protected int num_msgs_sent;
>>>>>>
>>>>>> No, this won't work... how would you do that ?
>>>>> Yes, this is the annotation syntax I had in mind, though, I was thinking
>>>>> about more high-level abstraction what's happening than just marking
>>>>> down injection points.
>>>>> Such as
>>>>>
>>>>> @ReceivedData
>>>>> public void receive(@From Address sender, byte[] data, int offset, @Size
>>>>> int length) {...}
>>>>>
>>>>> @ProcessingMessage
>>>>> protected void passMessageUp(@Message msg, ...) { ... }
>>>>>
>>>>> @ProcessingBatch
>>>>> protected void deliverBatch(@Batch MessageBatch batch)  { ... }
>>>>>
>>>>>
>>>>>> I don't really like this, on a general principle: AOP should *not* have
>>>>>> to change the src code in order to work. And the fact of the matter is
>>>>>> that you won't be able to identify *all* injection points beforehand...
>>>>>> unless you want to sprinkle your code with annotations.
>>>>> I have to agree with the fact that AOP should not have to change source.
>>>>> I had a special case in mind, that is tied to JGroups inspection and
>>>>> offers a way the monitoring with zero overhead when the monitoring is
>>>>> not in place. There, you'd just conceptually describe what JGroups does.
>>>>>
>>>>>>
>>>>>>> I was already thinking about this in relation with Message Flow Tracer
>>>>>>> [1] (not working right now as the JGroups have changed since I was
>>>>>>> writing that)?
>>>>>> I took a quick look: nice !
>>>>>>
>>>>>> This is exactly what I meant. Should be some sort of rule base in a VCS,
>>>>>> to which support engineers add rules when they have a case which
>>>>>> requires it and they deem it to be generally useful.
>>>>>>
>>>>>> Re API changes: doesn't Byteman have functionality which can check a
>>>>>> rule set against a code base (offline), to find out incompatibilities ?
>>>>>> Something like a static rule checker ?
>>>>> Right, this is possible - but you won't find if you've added another
>>>>> place that should be checked (e.g. MFT has to determine whether now
>>>>> you're processing a whole batch, or message alone - when you add a
>>>>> functionality to grab some stored messages and start processing them, as
>>>>> you do in UNICASTx, you won't spot that automatically).
>>>>>
>>>>> Beyond that, there are many false positives. E.g. if you have a never
>>>>> terminating loop in Runnable.run(), there is no place to inject the AT
>>>>> EXIT code and Byteman complains.
>>>>>
>>>>> In the end, human intervention is always required.
>>>>>
>>>>> Radim
>>>>>
>>>>>>> Roman Macor is right now updating the rules and I was
>>>>>>> hoping that we could insert annotations into JGroups that would be used
>>>>>>> instead of the rules (I was already considering different AOP framework
>>>>>>> as Byteman does not allow AT EXIT to catch on leaving exceptions [2]).
>>>>>> Yes, I've also run into this before, not really nice.
>>>>>>
>>>>>>> Radim
>>>>>>>
>>>>>>> [1] https://github.com/rvansa/message-flow-tracer
>>>>>>> [2] https://issues.jboss.org/browse/BYTEMAN-237
>>>>>>>
>>>>>>> On 11/07/2014 01:21 PM, Bela Ban wrote:
>>>>>>>> Hi Radim,
>>>>>>>>
>>>>>>>> no I haven't. However, you can replace the thread pools used by JGroups
>>>>>>>> and use custom pools.
>>>>>>>>
>>>>>>>> I like another idea better: inject Byteman code at runtime that keeps
>>>>>>>> track of this, and *other useful stats as well*.
>>>>>>>>
>>>>>>>> It would be very useful to support if we could ship a package to a
>>>>>>>> customer that is injected into their running system and grabs all the
>>>>>>>> vital stats we need for a few minutes, then removes itself again and
>>>>>>>> those stats are then sent to use as a ZIP file.
>>>>>>>> The good thing about byteman is that it can remove itself without a
>>>>>>>> trace; ie. there's no overhead before / after running byteman.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07/11/14 09:31, Radim Vansa wrote:
>>>>>>>>> Btw., have you ever considered checks if a thread returns to pool
>>>>>>>>> reasonably often? Some of the other datagrids use this, though there's
>>>>>>>>> not much how to react upon that beyond printing out stack traces (but
>>>>>>>>> you can at least report to management that some node seems to be broken).
>>>>>>>>>
>>>>>>>>> Radim
>>>>>>>>>
>>>>>>>>> On 11/07/2014 08:35 AM, Bela Ban wrote:
>>>>>>>>>> That's exactly what I suggested. No config gives you a shared global
>>>>>>>>>> thread pool for all caches.
>>>>>>>>>>
>>>>>>>>>> Those caches which need a separate pool can do that via configuration
>>>>>>>>>> (and of course also programmatically)
>>>>>>>>>>
>>>>>>>>>> On 06/11/14 20:31, Tristan Tarrant wrote:
>>>>>>>>>>> My opinion is that we should aim for less configuration, i.e.
>>>>>>>>>>> threadpools should mostly have sensible defaults and be shared by
>>>>>>>>>>> default unless there are extremely good reasons for not doing so.
>>>>>>>>>>>
>>>>>>>>>>> Tristan
>>>>>>>>>>>
>>>>>>>>>>> On 06/11/14 19:40, Radim Vansa wrote:
>>>>>>>>>>>> I second the opinion that any threadpools should be shared by default.
>>>>>>>>>>>> There are users who have hundreds or thousands of caches and having
>>>>>>>>>>>> separate threadpool for each of them could easily drain resources. And
>>>>>>>>>>>> sharing resources is the purpose of threadpools, right?
>>>>>>>>>>>>
>>>>>>>>>>>> Radim
>>>>>>>>>>>>
>>>>>>>>>>>> On 11/06/2014 04:37 PM, Bela Ban wrote:
>>>>>>>>>>>>> #1 I would by default have 1 thread pool shared by all caches
>>>>>>>>>>>>> #2 This global thread pool should be configurable, perhaps in the
>>>>>>>>>>>>> <global> section ?
>>>>>>>>>>>>> #3 Each cache by default uses the gobal thread pool
>>>>>>>>>>>>> #4 A cache can define its own thread pool, then it would use this one
>>>>>>>>>>>>> and not the global thread pool
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think this gives you a mixture between ease of use and flexibility in
>>>>>>>>>>>>> configuring pool per cache if needed
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 06/11/14 16:23, Pedro Ruivo wrote:
>>>>>>>>>>>>>> On 11/06/2014 03:01 PM, Bela Ban wrote:
>>>>>>>>>>>>>>> On 06/11/14 15:36, Pedro Ruivo wrote:
>>>>>>>>>>>>>>>> * added a single thread remote executor service. This will handle the
>>>>>>>>>>>>>>>> FIFO deliver commands. Previously, they were handled by JGroups incoming
>>>>>>>>>>>>>>>> threads and with a new executor service, each cache can process their
>>>>>>>>>>>>>>>> own FIFO commands concurrently.
>>>>>>>>>>>>>>> +1000. This allows multiple updates from the same sender but to
>>>>>>>>>>>>>>> different caches to be executed in parallel, and will speed thing up.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do you intend to share a thread pool between the invocations handlers of
>>>>>>>>>>>>>>> the various caches, or do they each have their own thread pool ? Or is
>>>>>>>>>>>>>>> this configurable ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That is question that cross my mind and I don't have any idea what would
>>>>>>>>>>>>>> be the best. So, for now, I will leave the thread pool shared between
>>>>>>>>>>>>>> the handlers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Never thought to make it configurable, but maybe that is the best
>>>>>>>>>>>>>> option. And maybe, it should be possible to have different max-thread
>>>>>>>>>>>>>> size per cache. For example:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * all caches using this remote executor will share the same instance
>>>>>>>>>>>>>> <remote-executor name="shared" shared="true" max-threads=4.../>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * all caches using this remote executor will create their own thread
>>>>>>>>>>>>>> pool with max-threads equals to 1
>>>>>>>>>>>>>> <remote-executor name="low-throughput-cache" shared="false"
>>>>>>>>>>>>>> max-threads=1 .../>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * all caches using this remote executor will create their own thread
>>>>>>>>>>>>>> pool with max-threads equals to 1000
>>>>>>>>>>>>>> <remote executor name="high-throughput-cache" shared="false"
>>>>>>>>>>>>>> max-thread=1000 .../>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> is this what you have in mind? comments?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Pedro
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>>
>>>>>
>>>
>>>
>>
>> --
>> Bela Ban, JGroups lead (http://www.jgroups.org)
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

More information about the infinispan-dev mailing list