Re: [Hawkular-dev] RfC: Availability

Great write-up Heiko, let me try to intersperse the outcomes of the yesterday's watercooler into it so that we get the discussion started. tl;dr - your ideas match almost exactly what we discussed on the meeting ;) The basic outcome of the meeting IMHO was that availability is in its most generic form a computed value and that the feeds might not be able to decide whether something is up or down. Judging such thing on "logical" resources like "applications" must be handled at some other layer either by polling for other data on a schedule and reporting the findings back into Hawkular by a completely 3rd party "reporter" or maybe such reporters would be hooked on the bus to receive "realtime" info about events and be able to compute the avail reactively. Also, it was agreed that even the availability states are not something we can decide upfront. Maybe we'd provide a number of avail values with undefined meaning that only avail "reporter" would be able define or we could understand avail as "health" ranging from 0-100% and again let the avail reporter decide on a value. On metrics level, the agreement was to keep the "availability" endpoints to be able to distinguish it from "normal" metrics. But there also were suggestions to just be able to mark some metrics as run-length encoded and consider avail as an ordinary metric. Other subsystems need to distinguish between avail and other metrics. Especially inventory needs to be told which metric corresponds to an avail value for particular resource. On Friday, March 20, 2015 11:20:17 Heiko W.Rupp wrote: If the feed is able to report on a "state" of something, I imagine we need at least 3 states: UP, DOWN and UNKNOWN. I imagine that a feed in and of itself wouldn't be able to report more elaborate states like "degraded" or "half- functional" or "up but rest api under heavy load" or such like. I imagine there either being a reporter that would compute such more involved states somewhere "up in the pipeline" or there could be some default simple translation like UP = 100%, DOWN = 0%, UNKNOWN = UNKNOWN (btw. I think "unknown" should be implicitly possible on any metric when I think about it, because it is distinctively different from "not collected"). The problem with GOING_DOWN is that it seems to me to be a one time event rather than a value of a metric. The inventory has metrics and resources decoupled from each other with just an m:n relationship between them. As such, I think inventory will just mark one of such relationships with an "avail" flag. The tuples therefore will just be `< avail_id, report_time, state >` Yes, exactly. Further to this, I think we may want (on case by case basis, I think) to actually NOT store the reported value but only store the computed one. +1 I am not sure we even want to have ADMIN_DOWN as an explicit avail state. All we need to have is some kind of metric reporting that right now, a flag called ON_PURPOSE = true (this can be just another metric coming from an actual user for example or as you say it can be a property on a resource or even the avail metric itself in inventory (every entity in inventory can store arbitrary key- value pairs)). We still can have avail UP or DOWN reported but because we know that this is ON_PURPOSE, alerts may choose to react differently for example. +1 As with ADMIN_DOWN, this can be modeled in a number of ways. This implies a read on each store of new incoming data. Logically, it makes sense, by I assume that internally, metrics will do this differently like for example actually store the raw values for a short-ish period of time and aggregate them to run-length encoded periodically. I think this is not precise. Let me illustrate on another example: Incoming data: 1) <rid, t1, UP> 2) <rid, t2, UP> 3) <rid, t3, DOWN> I think this should be encoded as: <rid, UP, t1, t2> <rid, UNKNOWN, t2, t3> <rid, DOWN, t3, t3> I.e. I would explicitly encode the information about the impossibility of knowing the precise time of going down. This also means that you know the last time you saw a resource up. I would also not store the "null" for the last run-length encoded piece, because you actually don't know that the resource is still up. Instead if someone asks for avail at time t4 (before next avail report comes after t3), I would report "unknown, last seen "down" at t3). (We could also just not store the UNKNOWN altogether and only interpret it from the lack of data for that period of time) +1 if we encode the transition from up to down as I outlined above, the last heard time is the start time of the unknown between the up and down. +1 I personally am a fan of avail being a percentage rather than a simple boolean. That makes the goes up/down less intuitive but if we understand the percentage as "health" it might not be that bad to understand: alert on health < 50% +1