I'm currently sketching a design for a REST health check endpoint for Infinispan

Hey Vojtech! JMX and CLI integration sounds very interesting. I also like the idea of exposing log and exception dump. Thanks a lot for the input! Sebastian On Tue, Jul 26, 2016 at 9:00 AM, Vojtech Juranek <vjuranek(a)redhat.com&gt; wrote: > On Tuesday 26 July 2016 07:10:16 Sebastian Laskawiec wrote: > > I'm currently sketching a design for a REST > > health check endpoint for Infinispan > > if it's not too broad, I'd include also various information about the > cluster > - e.g. number of machines in the cluster, recent exceptions in the log (or > dump of N lines of log) etc. If would be useful at least for testing > purposes > so that we won't have to gather various information via JMX and CLI > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hey Wolf! Technically it's possible but I'm not sure if we should do this. I think this is a responsibility of monitoring tools (e.g. Splunk, Kibana or even Zabbix). Thanks Sebastian On Tue, Jul 26, 2016 at 10:21 AM, Wolf Fink <wfink(a)redhat.com <mailto:wfink@redhat.com>> wrote: Do we expose historical data for the cluster view. Often it is important to see whether there are view changes, rebalancing and unexpected leave/merge events where nodes are kicked by JGroups. Having special entries for controlled view change and sudden view changes might be good On Tue, Jul 26, 2016 at 10:06 AM, Sebastian Laskawiec <slaskawi(a)redhat.com <mailto:slaskawi@redhat.com>> wrote: Hey Vojtech! JMX and CLI integration sounds very interesting. I also like the idea of exposing log and exception dump. Thanks a lot for the input! Sebastian On Tue, Jul 26, 2016 at 9:00 AM, Vojtech Juranek <vjuranek(a)redhat.com <mailto:vjuranek@redhat.com>> wrote: On Tuesday 26 July 2016 07:10:16 Sebastian Laskawiec wrote: > I'm currently sketching a design for a REST > health check endpoint for Infinispan if it's not too broad, I'd include also various information about the cluster - e.g. number of machines in the cluster, recent exceptions in the log (or dump of N lines of log) etc. If would be useful at least for testing purposes so that we won't have to gather various information via JMX and CLI _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Dear Community, I'd like to ask you for help. I'm currently sketching a design for a REST health check endpoint for Infinispan and I'm trying to imagine possible use cases.

Could you please give me a hand and tell me what functionalities are important for you? Would you like to be able to check status per-cache or maybe a red (not healthy), green (healthy), yellow (healthy, rebalance in progress) cluster status is sufficient? What kind of information do you expect to be there?

Hey Tristan! Comments inlined. Thanks Sebastian On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant <ttarrant(a)redhat.com <mailto:ttarrant@redhat.com>> wrote: On 26/07/16 07:10, Sebastian Laskawiec wrote: > Dear Community, > > I'd like to ask you for help. I'm currently sketching a design for a > REST health check endpoint for Infinispan and I'm trying to imagine > possible use cases. The health-check should be implemented as an MBean initially, with the ability to expose it via alternative implementations later. The server RESTful endpoint should be registered with the management interface via a special handler. Yes, I think it's a good idea. We could even use tools like Jolokia [1] to expose MBeans through REST interface (it can be added to standalone.conf to the bootstrap classpath). Alternatively we could use JDK embedded HTTP Server [2].

A cache and cachemanager's health is determined by a combination of parameters and we probably should allow for a user-pluggable checker. We already expose a number of statuses already, although obviously this would be an aggregate. Could you please elaborate more on that? How do we expose this information? Are you referring to Infinispan Stats [3]? I also though about supporting queries somehow. An imaginary example from the top of my head could look like the following: http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED //<-- This would return 200 OK if we have 3 nodes and given cache is in degraded mode. http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS //<-- Checks if we have 3 nodes and rebalance is in proress [3] https://github.com/infinispan/infinispan/tree/8.2.x/core/src/main/java/or... > > Could you please give me a hand and tell me what functionalities are > important for you? Would you like to be able to check status per-cache > or maybe a red (not healthy), green (healthy), yellow (healthy, > rebalance in progress) cluster status is sufficient? What kind of > information do you expect to be there? I wouldn't want this to be overly complex: a simple OK, KO should be sufficient. Additional detail may be optionally present, but not a requirement. I think we will need at least a 3rd state - yellow or something like this. This would mean that a rebalance is in progress of a node is joining/leaving. In other words - the cluster accepts requests but don't touch the nodes!

Just for clarification - if we used the management HTTP server - would it be possible to expose health endpoints in Library mode? I think the library use case might be also very important.

Hi Tristan! I've been investigating the possibility of integrating with WF monitoring endpoint and I found this article: [1]. If I understand the Infinispan and Wildfly integration bits correctly - all we need to do is to implement additional resources in [2]. This should make the new API available through CLI as well as through REST [1]. The biggest advantage of this solution is that one implementation solves both REST and CLI use cases at the same time. However there are some drawbacks too - we won't be able to support any custom queries (we are limited only to queries supported by WF bits) and the REST api will be a bit complicated to consume: * Imaginary examples based on [1]: o curl http://localhost:9990/management/subsystem=datagrid-infinispan/cache-cont... o curl http://localhost:9990/management/subsystem=datagrid-infinispan/cache-cont... * An example what ElasticSearch does [3]: o curl 'http://localhost:9200/_cluster/health?pretty=true' * An example what Spring Actuator does [4]: o curl 'http://localhost:8080/health' o curl 'http://localhost:8080/metrics' If I'm right - we will need to document this feature correctly since those URLs are not very intuitive. WDYT? Thanks Sebastian [1] https://docs.jboss.org/author/display/WFLY10/The+HTTP+management+API [2] https://github.com/infinispan/infinispan/tree/master/server/integration/i... [3] https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-h... [4] https://spring.io/guides/gs/actuator-service/ On Tue, Jul 26, 2016 at 10:34 AM, Tristan Tarrant <ttarrant(a)redhat.com <mailto:ttarrant@redhat.com>> wrote: On 26/07/16 10:24, Sebastian Laskawiec wrote: > Hey Tristan! > > Comments inlined. > > Thanks > Sebastian > > On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant <ttarrant(a)redhat.com <mailto:ttarrant@redhat.com> > <mailto:ttarrant@redhat.com <mailto:ttarrant@redhat.com>>> wrote: > > On 26/07/16 07:10, Sebastian Laskawiec wrote: > > Dear Community, > > > > I'd like to ask you for help. I'm currently sketching a design for a > > REST health check endpoint for Infinispan and I'm trying to imagine > > possible use cases. > The health-check should be implemented as an MBean initially, with the > ability to expose it via alternative implementations later. The server > RESTful endpoint should be registered with the management > interface via > a special handler. > > > Yes, I think it's a good idea. We could even use tools like Jolokia > [1] to expose MBeans through REST interface (it can be added to > standalone.conf to the bootstrap classpath). Alternatively we could > use JDK embedded HTTP Server [2]. No, for server we would not use Jolokia but rely on the management HTTP server (the one that handles port 9990 already). > > A cache and cachemanager's health is determined by a combination of > parameters and we probably should allow for a user-pluggable > checker. We > already expose a number of statuses already, although obviously this > would be an aggregate. > > > Could you please elaborate more on that? How do we expose this > information? Are you referring to Infinispan Stats [3]? > > I also though about supporting queries somehow. An imaginary example > from the top of my head could look like the following: > > http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED > //<-- This would return 200 OK if we have 3 nodes and given cache is > in degraded mode. > http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS > //<-- Checks if we have 3 nodes and rebalance is in proress > > [3] > https://github.com/infinispan/infinispan/tree/8.2.x/core/src/main/java/or... > > > > > > Could you please give me a hand and tell me what functionalities are > > important for you? Would you like to be able to check status > per-cache > > or maybe a red (not healthy), green (healthy), yellow (healthy, > > rebalance in progress) cluster status is sufficient? What kind of > > information do you expect to be there? > I wouldn't want this to be overly complex: a simple OK, KO should be > sufficient. Additional detail may be optionally present, but not a > requirement. > > > I think we will need at least a 3rd state - yellow or something like > this. This would mean that a rebalance is in progress of a node is > joining/leaving. In other words - the cluster accepts requests but > don't touch the nodes! Agreed. Tristan _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev