[infinispan-dev] Health check use cases

[Tristan Tarrant]

On 26/07/16 10:24, Sebastian Laskawiec wrote:
> Hey Tristan!
>
> Comments inlined.
>
> Thanks
> Sebastian
>
> On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant <ttarrant at redhat.com 
> <mailto:ttarrant at redhat.com>> wrote:
>
>     On 26/07/16 07:10, Sebastian Laskawiec wrote:
>     > Dear Community,
>     >
>     > I'd like to ask you for help. I'm currently sketching a design for a
>     > REST health check endpoint for Infinispan and I'm trying to imagine
>     > possible use cases.
>     The health-check should be implemented as an MBean initially, with the
>     ability to expose it via alternative implementations later. The server
>     RESTful endpoint should be registered with the management
>     interface via
>     a special handler.
>
>
> Yes, I think it's a good idea. We could even use tools like Jolokia 
> [1] to expose MBeans through REST interface (it can be added to 
> standalone.conf to the bootstrap classpath). Alternatively we could 
> use JDK embedded HTTP Server [2].

No, for server we would not use Jolokia but rely on the management HTTP 
server (the one that handles port 9990 already).

>
>     A cache and cachemanager's health is determined by a combination of
>     parameters and we probably should allow for a user-pluggable
>     checker. We
>     already expose a number of statuses already, although obviously this
>     would be an aggregate.
>
>
> Could you please elaborate more on that? How do we expose this 
> information? Are you referring to Infinispan Stats [3]?
>
> I also though about supporting queries somehow. An imaginary example 
> from the top of my head could look like the following:
>
> http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED 
> //<-- This would return 200 OK if we have 3 nodes and given cache is 
> in degraded mode.
> http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS 
> //<-- Checks if we have 3 nodes and rebalance is in proress
>
> [3] 
> https://github.com/infinispan/infinispan/tree/8.2.x/core/src/main/java/org/infinispan/stats
>
>
>     >
>     > Could you please give me a hand and tell me what functionalities are
>     > important for you? Would you like to be able to check status
>     per-cache
>     > or maybe a red (not healthy), green (healthy), yellow (healthy,
>     > rebalance in progress) cluster status is sufficient? What kind of
>     > information do you expect to be there?
>     I wouldn't want this to be overly complex: a simple OK, KO should be
>     sufficient. Additional detail may be optionally present, but not a
>     requirement.
>
>
> I think we will need at least a 3rd state - yellow or something like 
> this. This would mean that a rebalance is in progress of a node is 
> joining/leaving. In other words - the cluster accepts requests but 
> don't touch the nodes!
Agreed.

Tristan