From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; To: hawkular-dev(a)lists.jboss.org Sent: Monday, November 9, 2015 9:16:48 AM Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 Hi, Phase 0 is already implemented: - Metrics can receive data over HTTP - Metrics can publish inserted data to the bus - Webhooks can be created/configured via the Alerts API - Alerts is able to consume data from the bus

I can't see why we need to expose an endpoint in Metrics API to setup a set of conditions in Alerts. The Alerts API does the job.

Do you already know the use cases for Phase 1?

Regards, Thomas Le 06/11/2015 00:01, Stefan Negrea a écrit : > Hello Everybody, > > I want to open the discussion around the design and implementation for the > Hawkular Metrics and Hawkular Alerts integration. I created a document > with the scope and objectives for Phase 0. It is called Phase 0 because > this should be a proof of concept of the type of functionality to be > delivered in the future with a scope of getting early feedback on our > ideas. The goal is to deliver this phase in Hawkular Metrics 0.10.0, and > have something implemented in the next 2 weeks. > > I've had discussions with developers on both teams and updated the document > based on those discussions. My goal was to first refine the ideas to a > coherent state before publishing a first draft. I am now opening the > document and the discussion to the entire Hawkular community. > > The one point where we need additional help is highlighted in red with a > bigger font in the document. > > Please feel free to comment in the document, reply to this email, or > contact me (stefan_n) on #hawkular on freenode. If you do not have access > to the document please email me privately and will grant access. Any and > all the feedback is more than welcomed. > > > Link: > https://docs.google.com/document/d/1PwOGGysO0ppc2VPl80i4ye9pZI-Ugd4gObzQ5... > > > Thank you, > Stefan Negrea > > Software Engineer > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

Hello, That document uses what is already implemented as a base. Phase 0 (a prototype phase) should take the current implementation to the next level. All the things in the document where logical increments to what we have today. The goal was to have a quick Phase 0 to build something that we can get feedback on.

As for the requirements, there are no requirement and there will probably be very little requirements. There is no external entity to give us some requirements at this stage. If you consider that what is proposed in the document is wrong, I open to suggestions. However, I caution against inaction or slow action.

Whatever we do should follow two basic principles: 1) Fast phase, 2-3 weeks of development 2) Build something simple that enables more complicated features in the future

More replies inline below .... Thank you, Stefan ----- Original Message ----- > From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; > To: hawkular-dev(a)lists.jboss.org > Sent: Monday, November 9, 2015 9:16:48 AM > Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 > > Hi, > > Phase 0 is already implemented: > - Metrics can receive data over HTTP > - Metrics can publish inserted data to the bus > - Webhooks can be created/configured via the Alerts API > - Alerts is able to consume data from the bus That is not what the document states as goals for Phase 0. As the document is written what you described is a prerequisite for Phase 0.

> > I can't see why we need to expose an endpoint in Metrics API to setup a > set of conditions in Alerts. The Alerts API does the job. Disconnected APIs are not an integrated solutions. Just as an example, the are issues with the maturity of the APIs on both ends. Until we go through the exercise of trying to integrate the two in a user facing feature we will not see the differences or be able to provide something meaningful to users.

> > Do you already know the use cases for Phase 1? There are no requirements, and probably there will be no requirements in the next month (or even more). That is not the way to approach this.

Hello Thomas, So far there is no concrete proposal, only why things should not be done; which equates to inaction. I am not the most imaginative person on the planet so I might fail on that front, but I think the plan that I proposed had some cohesion and it was forward looking. Here is a good course of action: 1) Propose something new; completely new or partially new 2) Get the majority of the group (Metrics + Alerts teams) to agree on the path 3) Create the plan to accomplish it and lead the group to implement the plan

Thank you, Stefan Negrea ----- Original Message ----- > From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; > To: hawkular-dev(a)lists.jboss.org > Sent: Monday, November 9, 2015 10:43:26 AM > Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 > > Le 09/11/2015 16:48, Stefan Negrea a écrit : >> Hello, >> >> That document uses what is already implemented as a base. Phase 0 (a >> prototype phase) should take the current implementation to the next level. >> All the things in the document where logical increments to what we have >> today. The goal was to have a quick Phase 0 to build something that we can >> get feedback on. > > The use case detailed in the example is already possible with a Hawkular > server. > > I don't understand what taking the "current implementation to the next > level" means. > >> >> As for the requirements, there are no requirement and there will probably >> be very little requirements. There is no external entity to give us some >> requirements at this stage. If you consider that what is proposed in the >> document is wrong, I open to suggestions. However, I caution against >> inaction or slow action. >> > > The use case detailed in the document is fine. But there's no need to > prototype an integration if it already exists. > >> Whatever we do should follow two basic principles: >> 1) Fast phase, 2-3 weeks of development >> 2) Build something simple that enables more complicated features in the >> future >> > > I have yet to see a feature which requires to rebuild a new integration > mechanism. > >> >> More replies inline below .... >> >> >> Thank you, >> Stefan >> >> >> ----- Original Message ----- >>> From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; >>> To: hawkular-dev(a)lists.jboss.org >>> Sent: Monday, November 9, 2015 9:16:48 AM >>> Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 >>> >>> Hi, >>> >>> Phase 0 is already implemented: >>> - Metrics can receive data over HTTP >>> - Metrics can publish inserted data to the bus >>> - Webhooks can be created/configured via the Alerts API >>> - Alerts is able to consume data from the bus >> >> That is not what the document states as goals for Phase 0. As the document >> is written what you described is a prerequisite for Phase 0. > > From Google docs: > "This document has two parts. First are the details about how to > exchange data between the two services. And then, there is a description > of a simple feature that can showcase the integration." > > 1. Data exchange between the two services is already implemented. > 2. The user facing feature is already possible with a Hawkular server. > > So phase 0 is over. Period. > >> >>> >>> I can't see why we need to expose an endpoint in Metrics API to setup a >>> set of conditions in Alerts. The Alerts API does the job. >> >> Disconnected APIs are not an integrated solutions. Just as an example, the >> are issues with the maturity of the APIs on both ends. Until we go through >> the exercise of trying to integrate the two in a user facing feature we >> will not see the differences or be able to provide something meaningful to >> users. > > What does disconnected mean? That they are not implemented by the same > software component? In this case it's not a problem. > > The webhook API won't be more mature because it's re-written from Alerts > into Metrics. Maybe I missed your point. > > Integration of Alerts and Metrics has been demonstrated and used this > very afternoon at Devoxx by Clément in the vertx workshop. I'd bet no > attendee (other than Juca) did notice there was 2 software components > behind the service. > > Truth is the scenario described in the document lacks an integration > test. I believe this could be added to: > https://github.com/hawkular/hawkular/tree/master/modules/end-to-end-test > >> >>> >>> Do you already know the use cases for Phase 1? >> >> There are no requirements, and probably there will be no requirements in >> the next month (or even more). That is not the way to approach this. > > From my POV, the way to approach this is to write documentation and > tests for all the awesome features which already work but people > (including in our community) aren't aware of. > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

I guess what Stefan is trying to figure out is whether it makes sense and is useful to have Metrics use Alerts basically as a 3rd party tool, such that a potential Metrics client can focus on using Metrics and not worry about interacting with Alerts directly. The proposal to me looks like an attempt to identify one of possibly many Metrics "events" that a user may be interesting in setting up for notification, in this case via a webhook. So here we have proposed a MetricReported event (or was it MetricValueChanged?). I don't know exactly what other events would be useful, but perhaps there are a lot of Metrics happenings that a client may want to be notified of. It's not necessarily true that a client could do this by using the two tools independently, if in fact Metrics is adding hooks in its own processing to identify these "listenable events". But I don't get the feeling this is the proposal on the table, at least not for this first proposal. Instead, I think to handle MetricReported the idea may have been to just create a trigger that looks for the specific metric data coming in and to fire an alert. In which case it is perhaps not as useful to have Metrics set it up, but it's still maybe a step towards Metrics coming up with some sort of useful API that would leverage Alerts for its impl.

As for deployment, it seems to me that a prepackaged Metrics+Alerts+Bus, that could [eventually] be replicated/clustered, would be good. It could handle use cases where data is persisted and then pushed to the bus (like in hawkular), and then filtered and grabbed by alerts via a topic consumer. And it could handle REST interaction. And it could offer the ability to leverage co-located clients as well.

By the way, there is still the integration component I put together a while ago, that is completely independent of this discussion. It's an external alerter that is both a client of metrics and alerts, and generates alerts based on the results of aggregate queries performed against metrics. On 11/9/2015 12:48 PM, Thomas Segismont wrote: > Le 09/11/2015 18:10, Stefan Negrea a écrit : >> Hello Thomas, >> >> So far there is no concrete proposal, only why things should not be done; which equates to inaction. I am not the most imaginative person on the planet so I might fail on that front, but I think the plan that I proposed had some cohesion and it was forward looking. >> >> >> Here is a good course of action: >> 1) Propose something new; completely new or partially new >> 2) Get the majority of the group (Metrics + Alerts teams) to agree on the path >> 3) Create the plan to accomplish it and lead the group to implement the plan > I understand the goals of your proposal. I'm simply opposed to > re-writing what already exists, unless it is proven that it is broken. > > Given the stress you rightfully put on moving fast, improving the > existing system seems like a reasonable approach. > > Now maybe you didn't know the features are already there, but they are. > >> >> Thank you, >> Stefan Negrea >> >> ----- Original Message ----- >>> From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; >>> To: hawkular-dev(a)lists.jboss.org >>> Sent: Monday, November 9, 2015 10:43:26 AM >>> Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 >>> >>> Le 09/11/2015 16:48, Stefan Negrea a écrit : >>>> Hello, >>>> >>>> That document uses what is already implemented as a base. Phase 0 (a >>>> prototype phase) should take the current implementation to the next level. >>>> All the things in the document where logical increments to what we have >>>> today. The goal was to have a quick Phase 0 to build something that we can >>>> get feedback on. >>> The use case detailed in the example is already possible with a Hawkular >>> server. >>> >>> I don't understand what taking the "current implementation to the next >>> level" means. >>> >>>> As for the requirements, there are no requirement and there will probably >>>> be very little requirements. There is no external entity to give us some >>>> requirements at this stage. If you consider that what is proposed in the >>>> document is wrong, I open to suggestions. However, I caution against >>>> inaction or slow action. >>>> >>> The use case detailed in the document is fine. But there's no need to >>> prototype an integration if it already exists. >>> >>>> Whatever we do should follow two basic principles: >>>> 1) Fast phase, 2-3 weeks of development >>>> 2) Build something simple that enables more complicated features in the >>>> future >>>> >>> I have yet to see a feature which requires to rebuild a new integration >>> mechanism. >>> >>>> More replies inline below .... >>>> >>>> >>>> Thank you, >>>> Stefan >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; >>>>> To: hawkular-dev(a)lists.jboss.org >>>>> Sent: Monday, November 9, 2015 9:16:48 AM >>>>> Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 >>>>> >>>>> Hi, >>>>> >>>>> Phase 0 is already implemented: >>>>> - Metrics can receive data over HTTP >>>>> - Metrics can publish inserted data to the bus >>>>> - Webhooks can be created/configured via the Alerts API >>>>> - Alerts is able to consume data from the bus >>>> That is not what the document states as goals for Phase 0. As the document >>>> is written what you described is a prerequisite for Phase 0. >>> From Google docs: >>> "This document has two parts. First are the details about how to >>> exchange data between the two services. And then, there is a description >>> of a simple feature that can showcase the integration." >>> >>> 1. Data exchange between the two services is already implemented. >>> 2. The user facing feature is already possible with a Hawkular server. >>> >>> So phase 0 is over. Period. >>> >>>>> I can't see why we need to expose an endpoint in Metrics API to setup a >>>>> set of conditions in Alerts. The Alerts API does the job. >>>> Disconnected APIs are not an integrated solutions. Just as an example, the >>>> are issues with the maturity of the APIs on both ends. Until we go through >>>> the exercise of trying to integrate the two in a user facing feature we >>>> will not see the differences or be able to provide something meaningful to >>>> users. >>> What does disconnected mean? That they are not implemented by the same >>> software component? In this case it's not a problem. >>> >>> The webhook API won't be more mature because it's re-written from Alerts >>> into Metrics. Maybe I missed your point. >>> >>> Integration of Alerts and Metrics has been demonstrated and used this >>> very afternoon at Devoxx by Clément in the vertx workshop. I'd bet no >>> attendee (other than Juca) did notice there was 2 software components >>> behind the service. >>> >>> Truth is the scenario described in the document lacks an integration >>> test. I believe this could be added to: >>> https://github.com/hawkular/hawkular/tree/master/modules/end-to-end-test >>> >>>>> Do you already know the use cases for Phase 1? >>>> There are no requirements, and probably there will be no requirements in >>>> the next month (or even more). That is not the way to approach this. >>> From my POV, the way to approach this is to write documentation and >>> tests for all the awesome features which already work but people >>> (including in our community) aren't aware of. >>> _______________________________________________ >>> hawkular-dev mailing list >>> hawkular-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/hawkular-dev >>> >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev >> > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

and is useful to have Metrics use Alerts basically as a 3rd party tool, such that a potential Metrics client can focus on using Metrics and not worry about interacting with Alerts directly. The

I never said something is broken. The document is meant to be an iteration over what we have today. More precisely, contains two proposals for moving forward with an unified communication channel. And also there is a proposal for a new feature. I asked for feedback on those two communication methods to adopt one; with the goal to support the newly proposed feature. The current implementation is not broken for the current use cases (the integration in Hawkular). But that is not the point, and neither did I claim that is broken. It's not about fixing something that is broken but about creating something new. Just one example, a distributed environment with multiple Metrics and Alerts deployments will just not work with what is there today. And that was discussed thoroughly prior to making the document public.

On Nov 10, 2015, at 11:25 AM, Stefan Negrea <snegrea(a)redhat.com&gt; wrote: > > Can you elaborate on why what is in place today will not work with a > distributed environment with multiple Metrics and Alerts deployments? The user-facing feature proposed in the document will require request-reply communication with endpoints in Alerts. Also, there is the part about introducing a filtering mechanism for publishing only metrics of interest to Alerts. This also requires request-reply communication between the two services; the end-points are not implemented in any of the services but I do not see a reason not to make them in public REST API. So just for Phase 0, we will need a good number (5 or 6) of request-reply operations exposed over the communication channel. I am 100% sure in Phase 1 we will come up with some more features and will require more request-reply operations, so we might as well extend this to the all the public API in Phase 0. A goal for Phase 0 as stated in the document is to extend the public API currently in place in both service over the communication channel that we chose. There are two proposals in the document and both can achieve this: 1) Collocate one Metrics and one Alert instance in every single jvm (or container) and then make use of the existing public REST API over the local interface to communicate between the two components. a) No changes to the code in either service, just invoke the local public API interface of the other service when required to communicate.

b) Publishing of metrics insertion events will happen over the REST interface too; no longer use JMS. c) Distribution and load-balancing is done at the component level. For example: if Alerts needs to distribute the load across all the nodes deployed, it will be doing with its own mechanism. d) The challenge here is backpressure and availability of services. The expectation is that the collocated services will be able to handle the load that goes from one instance to the other.

2) Use JMS as the communication channel and then publish the REST API over JMS. And by "publishing the REST API over JMS", I mean making every single public end-point available over the JMS channel so there is no need in future phase to tinker with this. If we put in place a generic solution, we will never have to touch this again and we will have the side-effect of never having to think what is available between the two services; we will just focus on building features. a) No changes to the way metric insertion events are published, they will keep going over JMS. Future event types will go over JMS too. b) Metrics and Alerts components talk over JMS exclusively; the request-reply communication will most likely happen over temporary queue. c) The bus will facilitate some (or all) of the load distribution between components. c) The challenge with this approach is how to publish the public API over JMS. It's actually easy and I have a prototype, so it's just a matter of exploring more and finding a reasonable solution. We could also do a hybrid: collocated services, keep JMS for metric insertion events, everything else over local REST calls. After this long introduction, there are three main reasons the current solution needs improvements: 1) Addressability -> the current solution does not work in the distributed environment because there is no clear way how to access the public API of the services deployed. Let's say the installation spread across 5 containers. How can I make a public API call from a Metrics instance to an Alerts instance. There is no directory to know where the Alerts or Metrics instances are deployed.

2) Load distribution -> there is no clear way on how the load distribution works or who is responsible for it.

3) Availability of the existing public API -> There is no reason to implement a new API just for the purposes of communicating between the components. Given that we strive for this micro-service architecture, the one and single public API should be the main method for communicating between components for request-reply.

We might need to extend what we have but the public API should be front and centre. So if we use JMS or HTTP (or both, or UDP, or any other method), the public API interface should be available over all the channels. Sure there might be difference on how to make a request in JMS vs HTTP (one with temporary queues, and the other with direct http connections) but the functionality should be identical.

On Nov 10, 2015, at 12:57 PM, Stefan Negrea <snegrea(a)redhat.com&gt; wrote: The goal of this phase is to explore a few possibilities, come up with a decent solution, prototype and get feedback. I know there are flaws in all the proposals, otherwise we would have selected one already. I would like to refocus the discussion on working towards a concrete proposal to implement. >> >> There are two proposals in the document and both can achieve this: >> 1) Collocate one Metrics and one Alert instance in every single jvm (or >> container) and then make use of the existing public REST API over the >> local interface to communicate between the two components. >> a) No changes to the code in either service, just invoke the local public >> API interface of the other service when required to communicate. > > There should be no changes to the service implementations regardless of how > components communicate. When changes were recently made in H-Metrics to push > data onto the bus, there was no change in the service, just changes to > support forwarding data to other interested parties. Some sort of changes > will be necessary to forward data, regardless of whether we forward data > using the REST API, JMS, or something else. I would like to avoid as much as possible having fragmented public APIs for a component. With so many components, that will be our nemesis. Think about publishing those APIs in a 1.0 release and having to maintain them, deprecate them, and fix them. If we can keep a single unified public interface into a service, we will be able to spend time focused on features, not on maintenance. A single public API is much better than having special purpose APIs over different channels. I do not want to have to answer questions like, "is the tenant-id required when I insert metrics over JMS?", or "what is the request structure for querying metrics over UDP?". Now, realistically we will have to optimize for performance, hence idea of the "fireshose" as described in the document. But selecting to move something to the "firehose" should be a concious decision made for very specific reasons; the default should be public published API. > >> b) Publishing of metrics insertion events will happen over the REST >> interface too; no longer use JMS. >> c) Distribution and load-balancing is done at the component level. For >> example: if Alerts needs to distribute the load across all the nodes >> deployed, it will be doing with its own mechanism. >> d) The challenge here is backpressure and availability of services. The >> expectation is that the collocated services will be able to handle the >> load that goes from one instance to the other. > > Sorry, but I think that is a naive expectation. We are talking about a > distributed environment in which we want to be scalable. That to me means > expect and plan for failure not *if* but *when*, so please tell me how we > handle load/backpressure when do everything, including inter-component > communication, via REST. I suspect we will end up implementing functionality > already provided by messaging systems. But here is another more simpler to look based on the first proposal. Since there are multiple instances deployed, the expectation is that there will be a proxy in front of them. If the proxy is round-robin (or any other load balancing technique), that means any particular M+A instance will get hit randomly. If one of them fails, the user will have a failure returned, and then retry the request which will be routed to a different server. We add logic to note and count these failures in a M+A instance and shut it down after it reaches a threshold. From the discussions with Alerts folks, there are plans to implement a load distribution mechanism only for their purpose. Metrics is stateless at this stage and from the early designs will most like implement a different load distribution mechanism. So what does the bus add? Communication redundancy? We can have the user retry a failed attempt if a particular M+A instance is no longer viable. So the inconvenience here is the user might have to retry an operation. While we gain on the API front by using a single public API that is well known, documented, and tested. Trade-offs, trade-offs ... > >> >> 2) Use JMS as the communication channel and then publish the REST API over >> JMS. And by "publishing the REST API over JMS", I mean making every single >> public end-point available over the JMS channel so there is no need in >> future phase to tinker with this. If we put in place a generic solution, >> we will never have to touch this again and we will have the side-effect of >> never having to think what is available between the two services; we will >> just focus on building features. >> a) No changes to the way metric insertion events are published, they will >> keep going over JMS. Future event types will go over JMS too. >> b) Metrics and Alerts components talk over JMS exclusively; the >> request-reply communication will most likely happen over temporary queue. >> c) The bus will facilitate some (or all) of the load distribution between >> components. >> c) The challenge with this approach is how to publish the public API over >> JMS. It's actually easy and I have a prototype, so it's just a matter of >> exploring more and finding a reasonable solution. >> >> We could also do a hybrid: collocated services, keep JMS for metric >> insertion events, everything else over local REST calls. >> >> >> After this long introduction, there are three main reasons the current >> solution needs improvements: >> >> 1) Addressability -> the current solution does not work in the distributed >> environment because there is no clear way how to access the public API of >> the services deployed. Let's say the installation spread across 5 >> containers. How can I make a public API call from a Metrics instance to an >> Alerts instance. There is no directory to know where the Alerts or Metrics >> instances are deployed. > > Addressability is provided by the messaging system. There is no need for a > directory. You just to need to communicate with the messaging server/broker. > Beyond that there are a lot of features around addressability and routing > such as message selectors, message grouping, hierarchical topics, and more. Both solutions solve this in a form or another; it's about the trade-offs and what choices to make. > >> 2) Load distribution -> there is no clear way on how the load distribution >> works or who is responsible for it. > > Again, this is largely handled by the messaging system. Multiple consumers > take messages from a queue where each message corresponds to work to be > done. JMS solves load distribution for messaging, nothing more. But as a tread-off we will need to make the public API available over JMS if we are to go with the second proposal. > >> 3) Availability of the existing public API -> There is no reason to >> implement a new API just for the purposes of communicating between the >> components. Given that we strive for this micro-service architecture, the >> one and single public API should be the main method for communicating >> between components for request-reply. > > I do not think it is a given that we strive for a micro-service architecture. > It might make more sense in an OpenShift environment, but I don’t think it > necessarily does in general. > > >> We might need to extend what we have but the public API should be front and >> centre. So if we use JMS or HTTP (or both, or UDP, or any other method), >> the public API interface should be available over all the channels. Sure >> there might be difference on how to make a request in JMS vs HTTP (one >> with temporary queues, and the other with direct http connections) but the >> functionality should be identical. > > I don’t know that I agree with this. Suppose we decide to offer an API for > inserting metric data over UDP to support really high throughput situations. > Are you suggesting for example that the operations we provide via REST for > reading metric data should also be available via the UDP API? And since the > motivation for a UDP API is performance/throughput, we might event want to a > more compact request format than JSON. As I said early, unless this particular API is upgraded to "firehose", it should have an identical structure as the HTTP REST counter-part. The emphasis here is "structure", such as routing address, request content, and response content. For example, if tenant-id is required on one it should be required on both. > > Lastly and most importantly, if you want push for an alternative > communication mechanism between components in H-Metrics and H-Alerts, then > you push for the same across all of Hawkular because it does not make to > have to support two different mechanisms. I do not understand this contention point. The alternatives on the table are JMS and REST over HTTP. Both of these are in Hawkular today. All what the document proposes up to this point is to take one of those alternatives and deliver more functional on top of that foundation (I outlined in my previous email the differences between alternatives). I also mentioned in my previous email the hybrid approach: collocated services, keep JMS for metric insertion events, keep all the other inter-component communication over local REST calls (it's a mix between 1 and 2 above). That is exactly Hawkular with an extension to add needed pieces, such as event filtering and implementing the features purposed in the document. Do you see even this proposal a departure from the current Hawkular architecture?

The public APIs in Hawkular are the REST APIs. The REST APIs are the only “supported” public APIs.

From: "Lucas Ponce" <lponce(a)redhat.com&gt; To: "Discussions around Hawkular development" <hawkular-dev(a)lists.jboss.org&gt; Sent: Tuesday, November 10, 2015 12:58:06 PM Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 > The public APIs in Hawkular are the REST APIs. The REST APIs are > the only “supported” public APIs. As far as I remember, there was a discussion that Java API is also "supported". That was main reason we tried to unify naming of packaging. For example, all org.hawkular.<component>.api packages should be considered also "supportable". So, communication via Java API should be feasible.

_______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

> > > > So, communication via Java API should be feasible. > > That is an option as long as it works in a distributed environment (think > containers and different JVMs; would Java API work there? and how?) and the > same interface that is currently exposed on the public REST API is > available. > I'm not thinking as a general option. But I have a wildfly server with metrics.war and alerts.war and mycomponent.war as client under same deployment, why I can't use Java API and I need to use REST between them ? EJB clients are also possible but this was not introduced before and I don't want to do it. But in the first scenario, looking for a user that wants to add its code colocated, the Java API should be the easy one. _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

The hot transport right now is REST over HTTP.

Firstly, apologies if this thread is intended for Redhat staff only.

- therefore make the choice of easy integration. Abstract away how the

JMS solves load distribution for messaging, nothing more. But as a tread-off we will need to make the public API available over JMS if we are to go with the second proposal.

On Nov 10, 2015, at 1:09 PM, Lucas Ponce <lponce(a)redhat.com&gt; wrote: It is a little bit hard to jump into this long threads but interesting. I am going to describe my thoughts if that can help (or not) into this discussion. I think before to get into low level details, it is good to discuss perhaps in some high level motivations as ThomasS commented, and that might help to check in we are everyone in the same page. Today (if I am not wrong) we have three/four main scenarios where our efforts are driven: - A standalone use of Hawkular Metrics (mainly for OpenShift). - Hawkular, a integrated project of several components/subproject (bus,accounts,metrics,alerts,inventory,agent,ui,new ones). - Hawkular BTM (not sure if this will be consumed as part of Hawkular as add-on, or a variant). - Hawkular DataMining module (not sure if this will be consumed as part of Hawkular as add-on, or a variant). It happens as our components are decoupled and can be consumed separately, so it could be valid if the same user that is using metrics in standalone can add alerts too to work together, but probably that user is not interested in the full Hawkular solution as it has developed its own security, feeds and inventory frameworks (or any combination). So, for me, a legitimate question could be: "how easy it can be to put two components working together?". That I think is what we are discussing here. So, if a user can deploy a metrics.war and alerts.war in the deployments folder on a standard wildfly server, that could be interesting an attractive for everyone (or at least that I think is our main motivation for this discussion).

> If at the end, we reach to a conclusion that communication between > components needs an extra effort, I have my conclusion clear: we need to > go through Hawkular, not replicating things, or in other words, focus all > efforst in a single solution that can be used in several scenarios. As part of the migration to Wildfly 10, we will get rid of the ActiveMQ resource adapter and use the messaging capabilities offered in the container. So yes, when the migration is over, deploying Alerts + Metrics will be a matter of copying the archives to the deployments folder.

> After this long introduction, there are three main reasons the current solution needs improvements: > > > >1) Addressability -> the current solution does not work in the distributed environment because there is no clear way how to access the public API of the services deployed. Let's say the installation spread across 5 containers. How can I make a public API call from a Metrics instance to an Alerts instance. There is no directory to know where the Alerts or Metrics instances are deployed. Addressability is provided by the messaging system. There is no need for a directory. You just to need to communicate with the messaging server/broker. Beyond that there are a lot of features around addressability and routing such as message selectors, message grouping, hierarchical topics, and more.

> >2) Load distribution -> there is no clear way on how the load distribution works or who is responsible for it. Again, this is largely handled by the messaging system. Multiple consumers take messages from a queue where each message corresponds to work to be done.

> >3) Availability of the existing public API -> There is no reason to implement a new API just for the purposes of communicating between the components. Given that we strive for this micro-service architecture, the one and single public API should be the main method for communicating between components for request-reply. I do not think it is a given that we strive for a micro-service architecture. It might make more sense in an OpenShift environment, but I don’t think it necessarily does in general. > >We might need to extend what we have but the public API should be front and centre. So if we use JMS or HTTP (or both, or UDP, or any other method), the public API interface should be available over all the channels. Sure there might be difference on how to make a request in JMS vs HTTP (one with temporary queues, and the other with direct http connections) but the functionality should be identical. I don’t know that I agree with this. Suppose we decide to offer an API for inserting metric data over UDP to support really high throughput situations. Are you suggesting for example that the operations we provide via REST for reading metric data should also be available via the UDP API? And since the motivation for a UDP API is performance/throughput, we might event want to a more compact request format than JSON. Lastly and most importantly, if you want push for an alternative communication mechanism between components in H-Metrics and H-Alerts, then you push for the same across all of Hawkular because it does not make to have to support two different mechanisms.

On Nov 10, 2015, at 4:36 PM, Stefan Negrea <snegrea(a)redhat.com&gt; wrote: Thomas, I will ask the same thing I asked John earlier. Can you then please walk through how to implement the feature proposed in the document? I can see how it can be done in the proposals I put in the document and discussed so far. But looks like you might have a different alternative. Do you mind explaining the alternative and walking through the design and implementation? ----- Original Message ----- > From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; > To: hawkular-dev(a)lists.jboss.org > Sent: Tuesday, November 10, 2015 3:06:33 PM > Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 > > Le 10/11/2015 18:09, John Sanda a écrit : >>> After this long introduction, there are three main reasons the current >>> solution needs improvements: >>>> >>>> 1) Addressability -> the current solution does not work in the >>>> distributed environment because there is no clear way how to access the >>>> public API of the services deployed. Let's say the installation spread >>>> across 5 containers. How can I make a public API call from a Metrics >>>> instance to an Alerts instance. There is no directory to know where the >>>> Alerts or Metrics instances are deployed. >> Addressability is provided by the messaging system. There is no need for a >> directory. You just to need to communicate with the messaging >> server/broker. Beyond that there are a lot of features around >> addressability and routing such as message selectors, message grouping, >> hierarchical topics, and more. > > +1 > >> >>>> 2) Load distribution -> there is no clear way on how the load >>>> distribution works or who is responsible for it. >> Again, this is largely handled by the messaging system. Multiple consumers >> take messages from a queue where each message corresponds to work to be >> done. >> > > Right, load distribution is a feature of the messaging system. As for > HTTP load balancing, there is hardware and software dedicated to this, I > would avoid building an HTTP load balancer into the project. > >>>> 3) Availability of the existing public API -> There is no reason to >>>> implement a new API just for the purposes of communicating between the >>>> components. Given that we strive for this micro-service architecture, >>>> the one and single public API should be the main method for >>>> communicating between components for request-reply. >> I do not think it is a given that we strive for a micro-service >> architecture. It might make more sense in an OpenShift environment, but I >> don’t think it necessarily does in general. >> >> >>>> We might need to extend what we have but the public API should be front >>>> and centre. So if we use JMS or HTTP (or both, or UDP, or any other >>>> method), the public API interface should be available over all the >>>> channels. Sure there might be difference on how to make a request in JMS >>>> vs HTTP (one with temporary queues, and the other with direct http >>>> connections) but the functionality should be identical. >> I don’t know that I agree with this. Suppose we decide to offer an API for >> inserting metric data over UDP to support really high throughput >> situations. Are you suggesting for example that the operations we provide >> via REST for reading metric data should also be available via the UDP API? >> And since the motivation for a UDP API is performance/throughput, we might >> event want to a more compact request format than JSON. >> >> Lastly and most importantly, if you want push for an alternative >> communication mechanism between components in H-Metrics and H-Alerts, then >> you push for the same across all of Hawkular because it does not make to >> have to support two different mechanisms. > > > The public API (JSON over HTTP) is how the users must interact with the > Hawkular platform. It is the unique entry point for the users. > > > It's has been agreed during the project inception to make Hawkular > components talk to each other via the bus. It is not a trivial change to > break this assumption: everything the bus provides (see above) will need > to be re-implemented. > That is why I am opposed to the change, unless it is proven that it is a > limitation. > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

Thomas, I will ask the same thing I asked John earlier. Can you then please walk through how to implement the feature proposed in the document? I can see how it can be done in the proposals I put in the document and discussed so far. But looks like you might have a different alternative. Do you mind explaining the alternative and walking through the design and implementation? ----- Original Message ----- > From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; > To: hawkular-dev(a)lists.jboss.org > Sent: Tuesday, November 10, 2015 3:06:33 PM > Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 > > Le 10/11/2015 18:09, John Sanda a écrit : >>> After this long introduction, there are three main reasons the current >>> solution needs improvements: >>>> >>>> 1) Addressability -> the current solution does not work in the >>>> distributed environment because there is no clear way how to access the >>>> public API of the services deployed. Let's say the installation spread >>>> across 5 containers. How can I make a public API call from a Metrics >>>> instance to an Alerts instance. There is no directory to know where the >>>> Alerts or Metrics instances are deployed. >> Addressability is provided by the messaging system. There is no need for a >> directory. You just to need to communicate with the messaging >> server/broker. Beyond that there are a lot of features around >> addressability and routing such as message selectors, message grouping, >> hierarchical topics, and more. > > +1 > >> >>>> 2) Load distribution -> there is no clear way on how the load >>>> distribution works or who is responsible for it. >> Again, this is largely handled by the messaging system. Multiple consumers >> take messages from a queue where each message corresponds to work to be >> done. >> > > Right, load distribution is a feature of the messaging system. As for > HTTP load balancing, there is hardware and software dedicated to this, I > would avoid building an HTTP load balancer into the project. > >>>> 3) Availability of the existing public API -> There is no reason to >>>> implement a new API just for the purposes of communicating between the >>>> components. Given that we strive for this micro-service architecture, >>>> the one and single public API should be the main method for >>>> communicating between components for request-reply. >> I do not think it is a given that we strive for a micro-service >> architecture. It might make more sense in an OpenShift environment, but I >> don’t think it necessarily does in general. >> >> >>>> We might need to extend what we have but the public API should be front >>>> and centre. So if we use JMS or HTTP (or both, or UDP, or any other >>>> method), the public API interface should be available over all the >>>> channels. Sure there might be difference on how to make a request in JMS >>>> vs HTTP (one with temporary queues, and the other with direct http >>>> connections) but the functionality should be identical. >> I don’t know that I agree with this. Suppose we decide to offer an API for >> inserting metric data over UDP to support really high throughput >> situations. Are you suggesting for example that the operations we provide >> via REST for reading metric data should also be available via the UDP API? >> And since the motivation for a UDP API is performance/throughput, we might >> event want to a more compact request format than JSON. >> >> Lastly and most importantly, if you want push for an alternative >> communication mechanism between components in H-Metrics and H-Alerts, then >> you push for the same across all of Hawkular because it does not make to >> have to support two different mechanisms. > > > The public API (JSON over HTTP) is how the users must interact with the > Hawkular platform. It is the unique entry point for the users. > > > It's has been agreed during the project inception to make Hawkular > components talk to each other via the bus. It is not a trivial change to > break this assumption: everything the bus provides (see above) will need > to be re-implemented. > That is why I am opposed to the change, unless it is proven that it is a > limitation. > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

----- Original Message ----- > From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; > To: hawkular-dev(a)lists.jboss.org > Sent: Tuesday, November 10, 2015 4:09:15 PM > Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 > > - Start a Hawkular server > - Use the REST resources under /hawkular/alerts to create a trigger for > 'my_fancy_gauge' > - Use the REST resources under /hawkular/alerts to create an external > condition (my understanding is that such a condition will always > evaluate to true if data is received) > - Use the REST resources under /hawkular/alerts to create a webhook action > - Use the REST resources under /hawkular/metrics to publish data I was afraid of that answer. That was neither the spirit nor the intent of the proposal. > > > Le 10/11/2015 22:36, Stefan Negrea a écrit : >> >> >> Thomas, I will ask the same thing I asked John earlier. Can you then please >> walk through how to implement the feature proposed in the document? I can >> see how it can be done in the proposals I put in the document and >> discussed so far. But looks like you might have a different alternative. >> Do you mind explaining the alternative and walking through the design and >> implementation? >> >> >> >> ----- Original Message ----- >>> From: "Thomas Segismont" <tsegismo(a)redhat.com&gt; >>> To: hawkular-dev(a)lists.jboss.org >>> Sent: Tuesday, November 10, 2015 3:06:33 PM >>> Subject: Re: [Hawkular-dev] Hawkular Metrics + Hawkular Alerts - Phase 0 >>> >>> Le 10/11/2015 18:09, John Sanda a écrit : >>>>> After this long introduction, there are three main reasons the current >>>>> solution needs improvements: >>>>>> >>>>>> 1) Addressability -> the current solution does not work in the >>>>>> distributed environment because there is no clear way how to access the >>>>>> public API of the services deployed. Let's say the installation spread >>>>>> across 5 containers. How can I make a public API call from a Metrics >>>>>> instance to an Alerts instance. There is no directory to know where the >>>>>> Alerts or Metrics instances are deployed. >>>> Addressability is provided by the messaging system. There is no need for >>>> a >>>> directory. You just to need to communicate with the messaging >>>> server/broker. Beyond that there are a lot of features around >>>> addressability and routing such as message selectors, message grouping, >>>> hierarchical topics, and more. >>> >>> +1 >>> >>>> >>>>>> 2) Load distribution -> there is no clear way on how the load >>>>>> distribution works or who is responsible for it. >>>> Again, this is largely handled by the messaging system. Multiple >>>> consumers >>>> take messages from a queue where each message corresponds to work to be >>>> done. >>>> >>> >>> Right, load distribution is a feature of the messaging system. As for >>> HTTP load balancing, there is hardware and software dedicated to this, I >>> would avoid building an HTTP load balancer into the project. >>> >>>>>> 3) Availability of the existing public API -> There is no reason to >>>>>> implement a new API just for the purposes of communicating between the >>>>>> components. Given that we strive for this micro-service architecture, >>>>>> the one and single public API should be the main method for >>>>>> communicating between components for request-reply. >>>> I do not think it is a given that we strive for a micro-service >>>> architecture. It might make more sense in an OpenShift environment, but I >>>> don’t think it necessarily does in general. >>>> >>>> >>>>>> We might need to extend what we have but the public API should be front >>>>>> and centre. So if we use JMS or HTTP (or both, or UDP, or any other >>>>>> method), the public API interface should be available over all the >>>>>> channels. Sure there might be difference on how to make a request in >>>>>> JMS >>>>>> vs HTTP (one with temporary queues, and the other with direct http >>>>>> connections) but the functionality should be identical. >>>> I don’t know that I agree with this. Suppose we decide to offer an API >>>> for >>>> inserting metric data over UDP to support really high throughput >>>> situations. Are you suggesting for example that the operations we provide >>>> via REST for reading metric data should also be available via the UDP >>>> API? >>>> And since the motivation for a UDP API is performance/throughput, we >>>> might >>>> event want to a more compact request format than JSON. >>>> >>>> Lastly and most importantly, if you want push for an alternative >>>> communication mechanism between components in H-Metrics and H-Alerts, >>>> then >>>> you push for the same across all of Hawkular because it does not make to >>>> have to support two different mechanisms. >>> >>> >>> The public API (JSON over HTTP) is how the users must interact with the >>> Hawkular platform. It is the unique entry point for the users. >>> >>> >>> It's has been agreed during the project inception to make Hawkular >>> components talk to each other via the bus. It is not a trivial change to >>> break this assumption: everything the bus provides (see above) will need >>> to be re-implemented. >>> That is why I am opposed to the change, unless it is proven that it is a >>> limitation. >>> >>> _______________________________________________ >>> hawkular-dev mailing list >>> hawkular-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/hawkular-dev >>> >> >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev >> > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

inter-component communication, via REST. I suspect we will end up implementing functionality already provided by messaging systems.

> 2) Load distribution -> there is no clear way on how the load > distribution works or who is responsible for it. Again, this is largely handled by the messaging system. Multiple consumers take messages from a queue where each message corresponds to work to be done.

Lastly and most importantly, if you want push for an alternative communication mechanism between components in H-Metrics and H-Alerts, then you push for the same across all of Hawkular because it does not make to have to support two different mechanisms.

> Given the stress you rightfully put on moving fast, improving the > existing system seems like a reasonable approach. > > Now maybe you didn't know the features are already there, but they are. I know that you know that I know about the current state of the features. To reiterate, I am looking for a positive path forward.

opposed to re-writing what already exists, unless it is proven that it is broken.

Given the stress you rightfully put on moving fast, improving the existing system seems like a reasonable approach.