> Although it could be provided as a separate component, I am not sure we > have > any use cases where it would be used independent of the bus, so think it is > probably best to include it as additional functionality within the Bus > component. > > Agree/disagree? > In terms of archicture I would suggest to follow a similar architecture discussed for metrics/alerts/inventory. A main component with this persistence/Java API dependencies and then to a Listener/CDI/TBD (glue) project that integrate the component with the Bus, so then if needed, component could work without Bus dependencies. Perhaps we don't have an isolated use case without kettle/bus but I think that in that way we can unify the picture.

Hi, > > Hi Lucas > > ----- Original Message ----- > > > Although it could be provided as a separate component, I am not sure we > > > have > > > any use cases where it would be used independent of the bus, so think > > > it > > > is > > > probably best to include it as additional functionality within the Bus > > > component. > > > > > > Agree/disagree? > > > > > > > In terms of archicture I would suggest to follow a similar architecture > > discussed for metrics/alerts/inventory. > > > > A main component with this persistence/Java API dependencies and then to > > a > > Listener/CDI/TBD (glue) project that integrate the component with the > > Bus, > > so then if needed, component could work without Bus dependencies. > > Perhaps we don't have an isolated use case without kettle/bus but I think > > that in that way we can unify the picture. > > > > That sounds reasonable - although it is slightly different to how the > events/alerts discussion went (I believe) where I think the suggestion was > that events were persisted off the bus, including Alerts being published > and > stored, rather than locally storing - possibly my misunderstanding. > > However on reflection that would make the components more dependent upon > the > bus, so probably better to have a shared 'persistence' component, as you > suggest, which could be used by all of the other components to persist > information directly, and then use the bus to publish the data for further > processing. > > In terms of Elasticsearch - this 'persistence' component could initially > simply wrap Cassandra and then as we understand the query requirements, add > (possibly optional) support for Elasticsearch to do the advanced queries. > > Only component I am not sure about how it may use this shared 'persistence' > component is Inventory, as it uses Cassandra via Titan? > my points are two discuss perhaps in two layers: - the business aspect, to map better what an Event needs to be (what not) to identify the business implications in terms of lifecycle, where/when is generated, how interacts with alerts/metrics/possible inventory as well. It was an interesting meeting, as I think that the Event can be also mapped with log entries/audit/other messages, etc. But on this layer I would like to get the busines concept. - In the architecture aspect, I guess you are right, perhaps I didn't detail better my previous email, for example, in alerts, the architecture is to have a main engine that have a dependency with the cassandra cluster that can be shared (or not, this could be a future discussion) with metrics or other components, but the engine itself it has not a direct dependency with the bus, we add the bus dependency in an additional project, so we can have both alternatives: a use case where we can use the component in a standalone way or with the bus scenario, so it's a configuration/packaging option. My suggestion is that would be also interesting for EventsStore. Also, perhaps the ElasticSearch service can be shared as well for other components in the future. Regards, Lucas > Regards > Gary > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

Hi Gary, > > I believe currently the alerts engine does not subscribe to the bus for > events, so events need to be supplied to the alerts engine explicitly? > Today, alerts component listens from the bus for data alerts. Also, these data is "filtered" from metrics, in the meaning that not "all" data coming from metrics is insteresting for alerts, so in the "glue" we have a component to filter from one topic and put the data that is consumed for alerts. I guess that for Events we can do a similar approach. > Do we want the alerts engine to have that capability? i.e. as part of the > trigger definition, we offer the ability to define a bus > topic/queue/selector to feed the alert engine? > The trigger definition is more generic, it doesn't matter from alerts domain from which "source" a data is provided. The data model that alerts engine is expecting is here: https://github.com/hawkular/hawkular-alerts/tree/master/hawkular-alerts-a... Perhaps, we can add more data types if needed for inference processes. > Or should the "event store" allow a listener to be defined (in the glue) > that > could supply events to the alert engine - but then this slightly separates > the alert definition into two parts - the bit bridging the event store and > alert engine, and the trigger definition on how the event should be > processed. > I'm +1 to separate the bus integration layer aspect from the pure alerts inference. Today, we have the same separation and the "glue" code lives here: https://github.com/hawkular/hawkular-alerts/tree/master/hawkular-alerts-bus As we identified a "filtering" preliminar tasks that is not part of inference itself. In the other hand, having this separation we can maintain or goal to have an scenario with metrics+alerts and without bus dependencies. Regards, Lucas _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

From a BTM perspective, we will mainly be dealing with json on the bus, so its likely users would want to evaluate individual fields within the json structure. So string evaluation is ok for now, but I think we should plan for supporting more complex condition evaluation in the future.

Hi Gary, Probably restating much of the below... Alerts does integrate with the bus and listens on three topics. Two of which are specific to integrating with Hawkular Metrics, and one is relevant only to bus integration. Hawkular feeds will supply numeric and availability metrics and those will be published to the HawkularMetricData and HawkularAvailabilityData topics. H Alerts subscribes to them both. Because only a small subset of data persisted in metrics will be relevant to alerting (only those involved in defined trigger conditions), alerts filters out the irrelevant data in the listener code. It then translates the filtered metric data into the format suitable to alerts and forwards it for alert evaluation (via the Java API). Additionally, data can be published to the bus HawkularAlertData topic, and that data goes straight into alert processing, currently without filtering, although that may change. So, Events data could be published and picked up via the bus today by sending to the HawkularAlertData topic. Of course there is also a rest endpoint for sending in data. Alerting currently supports 3 types of "primitive" data: numeric, availability and string, which are used in the various condition types. But more likely we'd want something like you suggest, something analogous to the listener we have on HawkularMetricData, add a listener for published events, filter them as needed, and forward relevant events for evaluation. One question, I guess, is whether alerting on events requires a dedicated type, and dedicated condition types in alerting, or whether the current "primitives" are sufficient. For example, in RHQ, alerting on logging events could involve match on log level, the logfile path, and/or the actual log entry text. We don't have support for that kind of condition evaluation today in alerting. We'd rather have to distill it down to a single string datum and a regex test, for example. > Do we want the alerts engine to have that capability? i.e. as part of the trigger definition, we offer the ability to define a bus topic/queue/selector to feed the alert engine? So, I don't think we want to incorporate data sources/streams into our trigger definitions but rather, as Lucas says, insulate the engine and keep it cleanly separated from the bus. Keeping the reception/filtering/translating of event data in the hawkular (i.e. glue) code. Regards, Jay On 5/12/2015 11:50 AM, Gary Brown wrote: > Hi Lucas > > I'd just started looking at hawkular-alerts-bus :) > > Just wondering how complex the configuration may become in the glue, if it > is used to wire up the components but also potentially has to apply > filters - but we can see what is possible. > > Regards > Gary > > ----- Original Message ----- >> Hi Gary, >> >>> I believe currently the alerts engine does not subscribe to the bus for >>> events, so events need to be supplied to the alerts engine explicitly? >>> >> Today, alerts component listens from the bus for data alerts. >> >> Also, these data is "filtered" from metrics, in the meaning that not "all" >> data coming from metrics is insteresting for alerts, so in the "glue" we >> have a component to filter from one topic and put the data that is >> consumed >> for alerts. >> >> I guess that for Events we can do a similar approach. >> >> >>> Do we want the alerts engine to have that capability? i.e. as part of the >>> trigger definition, we offer the ability to define a bus >>> topic/queue/selector to feed the alert engine? >>> >> The trigger definition is more generic, it doesn't matter from alerts >> domain >> from which "source" a data is provided. >> >> The data model that alerts engine is expecting is here: >> >> https://github.com/hawkular/hawkular-alerts/tree/master/hawkular-alerts-a... >> >> Perhaps, we can add more data types if needed for inference processes. >> >>> Or should the "event store" allow a listener to be defined (in the glue) >>> that >>> could supply events to the alert engine - but then this slightly >>> separates >>> the alert definition into two parts - the bit bridging the event store >>> and >>> alert engine, and the trigger definition on how the event should be >>> processed. >>> >> I'm +1 to separate the bus integration layer aspect from the pure alerts >> inference. >> >> Today, we have the same separation and the "glue" code lives here: >> >> https://github.com/hawkular/hawkular-alerts/tree/master/hawkular-alerts-bus >> >> As we identified a "filtering" preliminar tasks that is not part of >> inference >> itself. >> >> In the other hand, having this separation we can maintain or goal to have >> an >> scenario with metrics+alerts and without bus dependencies. >> >> Regards, >> Lucas >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev >> > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

I briefly mentioned this in the F2F - a couple months ago I wrote an "audit" subsystem component prototype that basically does what we mentioned we want. The idea for this was it is a generic audit log - all other components would use it to log audit events (like "user failed to log in due to bad password" or "metric exceeded SLA" or whatever). We can obviously extend this to be ANY event, not just audit trails. For sure this is gonna miss some things we need, but essentially, this "audit" subsystem is simply a way to store "events" in a backend data store. It is what you guys have been talking about in this thread (I think, unless I didn't understand). Here's the code: https://github.com/jmazzitelli/hawkular-audit Just replace the word "Audit" with the word "Event" :) Here's what it does: 1) listens to a topic - anyone can send an "Audit Record" which consists of a timestamp (when it happened), a subsystem (who it happened to), a message string (what happened), and a Map<String, String> of details (more details on what happened). Think of a Subsystem as "Business Transaction Manager" or "Metrics" or "Alerts" or whatever we want. The Map can be anything - say, keyed on "transactionId" with the value the actual tx id. That's just an idea. We will probably extend that data structure to include more things if the above isn't enough. The point is, the "Audit Record" fully describes the event. Here's the AuditRecord (it is a JSON-encoded POJO): https://github.com/jmazzitelli/hawkular-audit/blob/master/hawkular-audit-... 2) once the message is received, it is stored using a storage mechanism. Right now, it supports storing it in a log file (org.jboss.logging) or a RDBMS (when I was testing, I was using H2). We'll want a Cassandra implementation. That's it. It is a very small component. It doesn't have a REST endpoint today, but we'll just add that like all our other components. It also has no notion of tenant Id, we'll need to add that too. Since this probably isn't going to be a component that must live standalone (like metrics) we won't have to play games with building two assemblies. We'll just always have a bus dependency on this component. I was going to slap a REST endpoint on here, and rename it from "audit" to "event" but I think I'm getting sick and jetlag is killing me :) I'll try to do something this week. Unless someone wants to take what I have and run with it. John Mazz _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

As per the watercooler call today, I renamed my prototype to "Events" ... take a look. Consider this a proposal for a starting point for an events component. Git clone this: https://github.com/jmazzitelli/hawkular-events and "mvn clean install" from the top (the default branch is "events" - that's the one you want, not master) Go to hawkular-events-distro/target, unzip the distro, and run wildfly*/bin/standalone.sh At this point, the events component is running as a standalone component in wildfly (remember this was a prototype written a while ago - its essentially a mini-kettle with just the bus and events components). Read the README for how you can send test events and look them up: https://github.com/jmazzitelli/hawkular-events/blob/events/README.adoc You'll notice the API jar is a separate artifact. All our other components that get deployed in the kettle can use the hawkular-events-api jar to send events. That API jar has this: https://github.com/jmazzitelli/hawkular-events/tree/events/hawkular-event... ... EventRecord is the domain object that you send (it is based on the bus framework, so its got full JSON support and can be used in the bus framework's listener APIs - which is why my MDBs are so small) - you send via the EventRecordProcessor (or a subclass if you wish to enhance it). Here's a couple of lines of code as an example that sends events (this is how the events test servlet sends events - essentially its how any component would do it) : https://github.com/jmazzitelli/hawkular-events/blob/events/hawkular-event... The events war just has some MDBs (aka bus listeners) and servlets (servlets are there mainly to test with). But the idea is that we have bus listeners for potentially different kinds of events, or listening to events over different queues (which is what I do here - LogOnlyEventsMDB takes events and logs them only - EventsMDB stores them in the backend datasource, currently just H2). Note that there is no REST interface (as I understand it, the events component is only for internal consumption - only kettle components will be using it - feeds will not be sending in events - so no need for REST AFAICS). Note the backend storage is RDBMS (H2) - we'd just have to implement a listener that stores to C*. Essentially do what this does: https://github.com/jmazzitelli/hawkular-events/blob/events/hawkular-event... So, its a standalone component. It utilizes the hawkular bus framework. It provides an independent API jar that other kettle components can use in order to send events to the events component via the bus. _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 05/08/2015 10:57 AM, Gary Brown wrote: > Hi > > As discussed yesterday, we need a general event store that can > persist relevant events off the bus. However we didn't decide where > this functionality should reside, whether in an existing component > or in a new component. > > Although it could be provided as a separate component, I am not > sure we have any use cases where it would be used independent of > the bus, so think it is probably best to include it as additional > functionality within the Bus component. > > Agree/disagree? With the micro-services approach in mind, I'd vote for a separate component for the event-store. Such component has a potential to grow in scope on the long term. > The other issues are: > > - how to determine which events should be persisted, as we may want > some events to trigger further processing by subscribers on the > bus, but not necessarily be stored for historic analysis. - for the > event store we will need to know the type of the event being > stored > > So one possible solution to both these issues is, if the 'type' of > the event is provided (possibly as a header value), then it will be > persisted? Could you give some examples of what "type" would be? I remember hearing "deployment" as one of the possible types, but would be interesting to think of some other cases.

I think there could also be a "group" or "category". This way, it would be possible to easily switch the persist/discard flag for a given set of types, depending on the current needs. For instance, "store all events for category 'application'" might be useful during an outage.

> Then within Elasticsearch, we can use the tenant/persona as the > Index (to truly segment the data) and the event type to ensure > different information types are partitioned/indexed appropriately. +1 - - Juca. _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

On 05/08/2015 11:25 AM, Gary Brown wrote: > Could you give some examples of what "type" would be? I remember > hearing "deployment" as one of the possible types, but would be > interesting to think of some other cases. >> Type could be anything, so a logical label such as "Log >> entry", "Alert", "BTMFragment", etc - just so that from an >> Elasticsearch perspective we could define a basic schema to >> identify how it should index the information for that type. >> Elasticsearch can dynamically create a schema based on the >> information it receives, but there are cases where you need to >> control how it processes the data (e.g. UUIDs) where you don't >> want it to analyse the field further. Then there could also be a "category" property, similar to the logging approach. For instance, BTM could send 50 events for a single BTM transaction, some with the FINEST category, some with CRITICAL, or even ERROR, and only a small portion of them (say, > INFO) would be persisted by default. In case of problems, it this could be changed to log all events. - Juca. _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJVTIOOAAoJECKM1e+fkPrXEnIIAJly6SWoPvr5IK+R8aeGO3YL QZU3ckbepRya1zWhlxoSqKi0V9qR0wIesQ5isK7R+iJvBX5xptdndBAfn2Odfc3X xeuhlUFHpZMpEdjGXtp0ozo7rIeGgWzunAK8oEEKEe+ydcVQNdKUPjvqzvO6TXlU MBEF3vvhFik0XqlmgXs9ZTg6nXTf46M7mbK1YzajPxOe5FmEtYdbb5WO4AHzRUD7 MGFQTZK4DKyGJVxj3+//FHmNOnhS2x99ae12e+Yw020NhTwFvEj+2HUn7M+Di6kp BOecMFxPw7klpdB6jKjwztvGFn86qWdh3c8AbDOum2dzBzFPVaag/HIGi+fKn0E= =J1zm -----END PGP SIGNATURE-----

> Although it could be provided as a separate component, I am not > sure we have any use cases where it would be used independent of > the bus, so think it is probably best to include it as additional > functionality within the Bus component. > > Agree/disagree? With the micro-services approach in mind, I'd vote for a separate component for the event-store. Such component has a potential to grow in scope on the long term.