[Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0

[Stefan Negrea]

> 
> Can you elaborate on why what is in place today will not work with a
> distributed environment with multiple Metrics and Alerts deployments?

The user-facing feature proposed in the document will require request-reply communication with endpoints in Alerts. Also, there is the part about introducing a filtering mechanism for publishing only metrics of interest to Alerts. This also requires request-reply communication between the two services; the end-points are not implemented in any of the services but I do not see a reason not to make them in public REST API.

So just for Phase 0, we will need a good number (5 or 6) of request-reply operations exposed over the communication channel. I am 100% sure in Phase 1 we will come up with some more features and will require more request-reply operations, so we might as well extend this to the all the public API in Phase 0. A goal for Phase 0 as stated in the document is to extend the public API currently in place in both service over the communication channel that we chose. 

There are two proposals in the document and both can achieve this:
1) Collocate one Metrics and one Alert instance in every single jvm (or container) and then make use of the existing public REST API over the local interface to communicate between the two components.
 a) No changes to the code in either service, just invoke the local public API interface of the other service when required to communicate.
 b) Publishing of metrics insertion events will happen over the REST interface too; no longer use JMS.
 c) Distribution and load-balancing is done at the component level. For example: if Alerts needs to distribute the load across all the nodes deployed, it will be doing with its own mechanism. 
 d) The challenge here is backpressure and availability of services. The expectation is that the collocated services will be able to handle the load that goes from one instance to the other.

2) Use JMS as the communication channel and then publish the REST API over JMS. And by "publishing the REST API over JMS", I mean making every single public end-point available over the JMS channel so there is no need in future phase to tinker with this. If we put in place a generic solution, we will never have to touch this again and we will have the side-effect of never having to think what is available between the two services; we will just focus on building features.
 a) No changes to the way metric insertion events are published, they will keep going over JMS. Future event types will go over JMS too.
 b) Metrics and Alerts components talk over JMS exclusively; the request-reply communication will most likely happen over temporary queue.
 c) The bus will facilitate some (or all) of the load distribution between components.
 c) The challenge with this approach is how to publish the public API over JMS. It's actually easy and I have a prototype, so it's just a matter of exploring more and finding a reasonable solution.

We could also do a hybrid: collocated services, keep JMS for metric insertion events, everything else over local REST calls.


After this long introduction, there are three main reasons the current solution needs improvements:

1) Addressability -> the current solution does not work in the distributed environment because there is no clear way how to access the public API of the services deployed. Let's say the installation spread across 5 containers. How can I make a public API call from a Metrics instance to an Alerts instance. There is no directory to know where the Alerts or Metrics instances are deployed.
2) Load distribution -> there is no clear way on how the load distribution works or who is responsible for it.
3) Availability of the existing public API -> There is no reason to implement a new API just for the purposes of communicating between the components. Given that we strive for this micro-service architecture, the one and single public API should be the main method for communicating between components for request-reply. We might need to extend what we have but the public API should be front and centre. So if we use JMS or HTTP (or both, or UDP, or any other method), the public API interface should be available over all the channels. Sure there might be difference on how to make a request in JMS vs HTTP (one with temporary queues, and the other with direct http connections) but the functionality should be identical.


Thank you,
Stefan 


> 
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>