[infinispan-dev] Health check use cases

[Tristan Tarrant]

We are a bit more complex than either ES and SA :)

First of all this needs to work for both standalone and domain modes, as 
well as support multiple cache containers. I believe it is possible to 
register an io.undertow.server.HttpHandler in the ManagementHttpServer 
to handle additional request types and have convenient aliases for the 
queries below.

Tristan

On 28/07/16 14:09, Sebastian Laskawiec wrote:
> Hi Tristan!
>
> I've been investigating the possibility of integrating with WF 
> monitoring endpoint and I found this article: [1].
>
> If I understand the Infinispan and Wildfly integration bits correctly 
> - all we need to do is to implement additional resources in [2]. This 
> should make the new API available through CLI as well as through REST [1].
>
> The biggest advantage of this solution is that one implementation 
> solves both REST and CLI use cases at the same time. However there are 
> some drawbacks too - we won't be able to support any custom queries 
> (we are limited only to queries supported by WF bits) and the REST api 
> will be a bit complicated to consume:
>
>   * Imaginary examples based on [1]:
>       o curl
>         http://localhost:9990/management/subsystem=datagrid-infinispan/cache-container=local?operation=health&recursive=true&json.pretty=1
>       o curl
>         http://localhost:9990/management/subsystem=datagrid-infinispan/cache-container=local/local-cache=default?operation=health&json.pretty=1
>   * An example what ElasticSearch does [3]:
>       o curl 'http://localhost:9200/_cluster/health?pretty=true'
>   * An example what Spring Actuator does [4]:
>       o curl 'http://localhost:8080/health'
>       o curl 'http://localhost:8080/metrics'
>
> If I'm right - we will need to document this feature correctly since 
> those URLs are not very intuitive. WDYT?
> Thanks
> Sebastian
>
> [1] https://docs.jboss.org/author/display/WFLY10/The+HTTP+management+API
> [2] 
> https://github.com/infinispan/infinispan/tree/master/server/integration/infinispan/src/main/java/org/jboss/as/clustering/infinispan/subsystem
> [3] 
> https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html
> [4] https://spring.io/guides/gs/actuator-service/
>
> On Tue, Jul 26, 2016 at 10:34 AM, Tristan Tarrant <ttarrant at redhat.com 
> <mailto:ttarrant at redhat.com>> wrote:
>
>     On 26/07/16 10:24, Sebastian Laskawiec wrote:
>     > Hey Tristan!
>     >
>     > Comments inlined.
>     >
>     > Thanks
>     > Sebastian
>     >
>     > On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant
>     <ttarrant at redhat.com <mailto:ttarrant at redhat.com>
>     > <mailto:ttarrant at redhat.com <mailto:ttarrant at redhat.com>>> wrote:
>     >
>     >     On 26/07/16 07:10, Sebastian Laskawiec wrote:
>     >     > Dear Community,
>     >     >
>     >     > I'd like to ask you for help. I'm currently sketching a
>     design for a
>     >     > REST health check endpoint for Infinispan and I'm trying
>     to imagine
>     >     > possible use cases.
>     >     The health-check should be implemented as an MBean
>     initially, with the
>     >     ability to expose it via alternative implementations later.
>     The server
>     >     RESTful endpoint should be registered with the management
>     >     interface via
>     >     a special handler.
>     >
>     >
>     > Yes, I think it's a good idea. We could even use tools like Jolokia
>     > [1] to expose MBeans through REST interface (it can be added to
>     > standalone.conf to the bootstrap classpath). Alternatively we could
>     > use JDK embedded HTTP Server [2].
>
>     No, for server we would not use Jolokia but rely on the management
>     HTTP
>     server (the one that handles port 9990 already).
>
>     >
>     >     A cache and cachemanager's health is determined by a
>     combination of
>     >     parameters and we probably should allow for a user-pluggable
>     >     checker. We
>     >     already expose a number of statuses already, although
>     obviously this
>     >     would be an aggregate.
>     >
>     >
>     > Could you please elaborate more on that? How do we expose this
>     > information? Are you referring to Infinispan Stats [3]?
>     >
>     > I also though about supporting queries somehow. An imaginary example
>     > from the top of my head could look like the following:
>     >
>     >
>     http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED
>     > //<-- This would return 200 OK if we have 3 nodes and given cache is
>     > in degraded mode.
>     >
>     http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS
>     > //<-- Checks if we have 3 nodes and rebalance is in proress
>     >
>     > [3]
>     >
>     https://github.com/infinispan/infinispan/tree/8.2.x/core/src/main/java/org/infinispan/stats
>     >
>     >
>     >     >
>     >     > Could you please give me a hand and tell me what
>     functionalities are
>     >     > important for you? Would you like to be able to check status
>     >     per-cache
>     >     > or maybe a red (not healthy), green (healthy), yellow
>     (healthy,
>     >     > rebalance in progress) cluster status is sufficient? What
>     kind of
>     >     > information do you expect to be there?
>     >     I wouldn't want this to be overly complex: a simple OK, KO
>     should be
>     >     sufficient. Additional detail may be optionally present, but
>     not a
>     >     requirement.
>     >
>     >
>     > I think we will need at least a 3rd state - yellow or something like
>     > this. This would mean that a rebalance is in progress of a node is
>     > joining/leaving. In other words - the cluster accepts requests but
>     > don't touch the nodes!
>     Agreed.
>
>     Tristan
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev