On 02/13/2015 11:18 PM, Michael Burman wrote: Recording all datapoints doesn't give you more information on what happened between 2 recordings though. But I get your point on the "inability to prove that monitoring reported up on certain time when it wasn't responsive". I didn't see availability recording as SLA proofs though. You can't prove that system is up and running at a special point in time, you can only prove over a period of time if you made enough checks (interval lesser than the period of time of the SLA). Thomas

I will add the following to the discussion: If past history with customers of JON/users of RHQ/salespeople is any indication, we will get requests to support collecting availability every second (if not every sub-second) even AFTER telling people, documenting, and shouting from the mountaintop that we are not a realtime profiler. So, do we have the throughput/infrastructure that can support one "UP" datapoint coming in every minute for N resources. Do we want to support this? Resource #1: 11:00:01 UP Resource #1: 11:00:02 UP Resource #1: 11:00:03 UP Resource #1: 11:00:04 UP ...56 datapoints later.... Resource #1: 11:01:00 UP Resource #1: 11:01:01 UP Resource #1: 11:01:02 UP ... and on ... and repeat for Resource #3, #5, etc (not every resource, but many of them). Perhaps we store data like this, but only quickly aggregate them (say, every hour, aggregate this data into a tuple like "UP, 11:00:01, 11:01:02" where that is "avail state, begin time, end time"). I also find the pattern of the typical avail state of resources should be considered when designing the best storage schema for availability data (i.e. changes are very infrequent in between large amounts of time where the avail state is the same - "UP UP UP UP UP ... UP DOWN DOWN DOWN DOWN ..." - what's the best way to store that?). Storing just state changes has problems as we know (e.g. with alerts - we can only alert on state changes, like "Going UP" or "Going DOWN" not to mention what about admin states like "maintenance". ----- Original Message -----

On 01/22/2015 07:48 PM, John Sanda wrote: It should be reasonable ;) IMHO typically one would be happy with a check every 20s/30s or even a minute if the cost is "low", ie: the cost of the monitoring infrastructure is a small fraction of the infrastructure itself. We need to scale horizontally for processing data and storage (adding Cassandra nodes ?), reaching 1s support won't come without additional infrastructure cost. For sub-second support, I would question the need to store that information, I don't think we should look into this now. Thomas

On Tuesday, February 17, 2015 11:52:28 AM Heiko W.Rupp wrote: Do you envision these being generated on the feed-side or server-side? Currently, all we plan to do is advise a scheme for getting unique resource ids on the feed-side (almost) without server intervention. The feed only needs to remember its server-assigned ID. From there on, it can use the naming scheme of (something like): feed://<feedId>/resource/<resourceId> where resource ID is generated on the feed side (and can be numeric, name, whathever, we don't care). (https://issues.jboss.org/browse/HWKINVENT-12)

Hi, We already have dependency to Google's Guava, so there's a well tested cache implementation (with filling strategies and eviction strategies) already. No need to reinvent this one ;) - Micke On 17.02.2015 12:52, Heiko W.Rupp wrote:

There was a long IRC chat on this topic yesterday, for thread completeness I'm adding it here... (11:53:43 AM) jshaughn: general question, for feeds sending various types of MetricData, let's say NumericData and Availability (the only two I know exist currently), can the data be mixed in a single push, or can we assume that it will arrive as two different requests, one for each type? (11:54:15 AM) jshaughn: stefan_n, jsanda maybe you have an idea on that already? (11:55:22 AM) jsanda: jshaughn: it's separate (11:55:47 AM) jsanda: jshaughn: but we can certainly change that if it makes sense to do so (11:56:56 AM) jshaughn: jsanda: no, I think it may be better to have it separate (11:57:49 AM) jshaughn: it will be easier to parse out the json consistently if it's separate. also, it may require different processing. Given what we have been discussing in that avail thread (11:58:37 AM) jshaughn: perhaps H metrics should (optionally) provide caching of avail data, such that it gets stored changes-only. (11:58:58 AM) jshaughn: not sure if that should be at the metrics level or higher up (11:59:48 AM) jshaughn: but if metrics is going to natively handle avail data, then perhaps that should be a metrics feature. But I'm not sure about the whole ability of a cluster-wide cache. (12:00:01 PM) jshaughn: the other option is the idea of aggregates (12:00:14 PM) jsanda: i would consider it more an implementation detail than a feature (12:00:39 PM) jshaughn: I'm not sure, because users may or may not want changes-only storage (12:01:02 PM) jsanda: your mention of a cluster-wide cache is timely considering i brought up spark earlier today on the mailing list (12:01:08 PM) jshaughn: then again, we don't have to be that flexible, we could decide changes-only is the way it will be (12:01:15 PM) jsanda: users could get changes only either way (12:01:44 PM) jshaughn: right, the question is whether it's worthwhile to perform the storage of all the data points (12:02:28 PM) jshaughn: when 99% of the avail data is UP, it seems kind of useless to store every DP (12:02:44 PM) jsanda: data would be compressed on disk, which i think would make a substantial difference in terms of disk foot print (12:02:56 PM) jsanda: i haven't thought enough about it either way (12:03:02 PM) jshaughn: it's true what Micke said, about being able to prove the Up-ness was actually measured, but I'm not convinced it's necessary (12:04:00 PM) jshaughn: jsanda: not sure it matters footprint wise, but it is potentially more processing, more fetch time, aggregation time, etc (12:04:09 PM) jshaughn: time that could be spent doing other things (12:04:48 PM) jsanda: only storing changes introduces its own complexity (12:04:53 PM) jshaughn: fetch and store would be the other option (12:05:05 PM) jsanda: i don'nt think so (12:05:47 PM) jsanda: maybe if we are storing avail in postgres, but definitely not if it is being stored in C* (12:06:08 PM) jshaughn: I didn't think so, either, I thought I actually saw you propose that in the thread (12:06:46 PM) jsanda: it's also possible to do a combination (12:06:52 PM) tsegismont: how do we model absence of value when storing changes only? (12:06:56 PM) jsanda: we store the changes (12:07:08 PM) jsanda: in addition to all the raw values (12:07:19 PM) jsanda: which we could roll up (12:07:54 PM) jsanda: and purge the raw as aggressively as we want (12:08:42 PM) jshaughn: tsegismont: we don't really, I think. Unless we want to keep a counter of how many repeats we got in an interval (you lose the timestamps but could likely assume that in most cases they came in at regular intervals) (12:09:03 PM) jsanda: tsegismont: one way might be for a job to run on the server side and performs the check for a value being reported within some window (12:09:26 PM) tsegismont: jsanda which means giving intelligence to metrics (12:09:38 PM) jshaughn: what exactly is the need? (12:09:39 PM) jsanda: yes (12:09:40 PM) tsegismont: which shouldn't take care of that, IMHO (12:10:00 PM) jsanda: what do you mean by intelligence? (12:10:15 PM) tsegismont: " a job to run on the server side and performs the check for a value being reported within some window" (12:10:19 PM) jsanda: we already have use cases for different jobs that will have to run (12:10:27 PM) jshaughn: aggregation could potentially keep the count of the raw metrics collected during an interval of unchanged avail (12:10:53 PM) tsegismont: metrics itself does know how often a metric/avail should be reported (12:11:06 PM) tsegismont: it takes timestamped-reports, that's it (12:11:17 PM) jsanda: why can't it know? (12:12:02 PM) jsanda: just like metrics needs to when data retention settings or when various rollups need to be computed (12:12:29 PM) tsegismont: jsanda, is it stored in metadata? (12:13:34 PM) jsanda: it isn't stored anywhere because we currently aren't doing anything with availability (12:13:39 PM) jsanda: but it could be stored (12:14:04 PM) tsegismont: Let's say you have an avail report looking like (12:14:16 PM) tsegismont: 5:43:12 UP (12:14:31 PM) tsegismont: 5:43:32 UP (12:14:39 PM) tsegismont: 5:43:35 DOWN (12:15:43 PM) tsegismont: Then you're able to say: it went DOWN at 5:43:35, and last up report was 3 seconds before (12:15:50 PM) tsegismont: now if you store only differences (12:16:17 PM) tsegismont: and have a report like: (12:16:32 PM) tsegismont: 3:43:12 UP (12:16:33 PM) tsegismont: 5:43:35 DOWN (12:17:01 PM) tsegismont: You can only say: it went DOWN at 5:43:35, and was up since 3:43:12 (12:17:51 PM) tsegismont: To make a long story short: if you store things like this, you lose information (12:18:01 PM) tsegismont: but maybe it's not the intention? (12:18:29 PM) jsanda: your example makes sense (12:18:53 PM) jsanda: and i agree that there may be value in storing all values (12:19:19 PM) jsanda: i haven't formed an opinion one way or the other though (12:19:26 PM) jsanda: need to think about it more (12:19:42 PM) jsanda: obviously pros/cons with each (12:22:28 PM) jshaughn: if you stored 3:43:12 UP (N), 43:35 DOWN (2) You'd know that you had 1 UP in between. If there was some sort of aggregation of the raw data this may be feasible. You wouldn't know the exact times, but could likely assume the reports came in at regular intervals (12:23:17 PM) mazz: what's the (N) and (2) mean? (12:23:32 PM) mtho11: if for this first MVP we just store the changes I think it will make our lives easier to focus on other higher value things (not to mention we could change this in phase 2) (12:24:02 PM) mtho11: we wouldnt even have do any aggregations on that data most likely (12:24:13 PM) jshaughn: mazz, that would be the count of raw avails contributing the the rollup (12:24:35 PM) jshaughn: probably a dumb idea, but an idea nonetheless (12:25:01 PM) jsanda: i don't think storing only changes is necessarily easier (12:25:08 PM) tsegismont: jsanda, +1 (12:25:10 PM) jsanda: in fact, i think it is harder (12:25:12 PM) jsanda: a lot (12:25:14 PM) jshaughn: so if you have 10000 UPs in a rown before it go's DOWN you'd have DOWN time (10000) (12:25:35 PM) jsanda: because we need to introduces a caching/clustering layer, i.e., spark (12:25:42 PM) jshaughn: storing changes-only is certainly harder. (12:25:44 PM) jshaughn: As it stands right now I think we have to just go with storing all values, leaving it up to the consuming apps to do anything other than pass it through in raw form. (12:26:08 PM) jsanda: and i also think it is fair to say it is an implementation detail (12:26:21 PM) tsegismont: <jsanda> in fact, i think it is harder +1 (12:26:26 PM) mazz: right, the reason why today RHQ does it like it does it (changes only) is because we would have clobbered the DB with all those avail rows (12:26:38 PM) mazz: with all basically the same UP value :) (12:27:20 PM) mazz: but if C* has better performance for that type of data, then we don't need to worry about it (12:28:13 PM) jshaughn: I still don't see it as an implementation detail (12:28:45 PM) jsanda: i think the issue won't be so much about write performance as much about the disk foot print and the cpu needed to aggregate and purge the raw values (12:28:50 PM) jshaughn: because queries will return different data given the different approaches. (12:29:02 PM) mtho11: all long as you can provide the UI with just the changes that is all I care about :) (12:29:10 PM) jsanda: with that said, there is additional overhead as well for implementing a solution to only store changes (12:29:21 PM) jshaughn: mtho11: right, that is why it's not an impl detail, I think (12:29:37 PM) mtho11: great (12:29:51 PM) jsanda: jshaughn: why would the queries return different data? (12:30:11 PM) jsanda: if we have all the values, we can return only the changes (12:30:14 PM) jsanda: right? (12:30:18 PM) jshaughn: if it's changes only raw data would return repeats (and likely a lot of them), if not, it wouldn't. (12:30:39 PM) mazz: so don't return all the raw data (12:30:43 PM) jsanda: right (12:30:49 PM) jsanda: we filter the repeats out (12:30:51 PM) jshaughn: then why store it? (12:31:02 PM) jsanda: do the aggregation at query time (12:31:02 PM) mazz: the storage is an impl detail because under the covers, we just filter out what the API returns (12:31:08 PM) jsanda: yep (12:31:10 PM) jsanda: easy peasy (12:31:13 PM) mazz: why store it? (12:31:19 PM) mazz: perhaps because it is faster? (12:31:22 PM) mazz: I don't know (12:31:25 PM) mazz: that would be a reason though (12:31:33 PM) jshaughn: the argument above was that the details, the repeated values, were valuable (12:31:34 PM) jsanda: because it is easier than only storing changes (12:31:42 PM) mazz: if its faster to just take the data and store it rather than do some pre-filtering to see if we need to insert it or not (12:31:51 PM) jsanda: and the example that tsegismont provided is a good one (12:31:58 PM) jshaughn: I think Metrcs would need to have services to return raw and aggregated (12:33:05 PM) jsanda: we have a stream of values like (12:33:14 PM) jshaughn: tsegismont: and gaYak were saying the same thing. The question is whether it's worth the extra storage work. But it's moot if we don't have a good idea of how to distill raw avail to avail ranges (12:33:30 PM) jshaughn: on the way in (12:33:50 PM) jsanda: that extra storage work could be less work than only storing the changes (12:33:59 PM) jshaughn: personally, I'm OK with storing it all, at least to start with (12:34:22 PM) jsanda: maybe it won't be, but i think it will be more work, more resources, etc (12:34:35 PM) jshaughn: well, if we had a global cache it wouldn't be hard to filter duplicates (12:34:51 PM) tsegismont: We don't need to keep that level of details for years (12:35:01 PM) jsanda: right (12:35:17 PM) jsanda: as i said, i think we could purge the data pretty aggressively (12:35:32 PM) jshaughn: so what is the Spark angle? (12:36:14 PM) jsanda: distributed processing/caching that is fault tolerant and persistent (12:36:45 PM) jsanda: this is a perfect use case (12:37:26 PM) jsanda: i'm not sure that the benefits outweigh the added complexity though (12:37:28 PM) mtho11: jsanda: doesnt purging the data aggressively cause Garbage Collection issues? (12:37:54 PM) mtho11: availability could be on a massive scale (12:38:15 PM) jsanda: not that i'm aware of (12:39:08 PM) jsanda: we could consider using date tiered compaction which can drop an entire sstable at a time (12:39:24 PM) jsanda: so the purging would be more coarse grained (12:39:25 PM) mtho11: more records get purged than get saved (12:39:29 PM) jshaughn: one side-note, I'm hoping that actually a lot of avail filtering is done by the feeds. Meaning, I think they should report avail only on critical resources, not *all* of them like in RHQ. That was pretty useless if you ask me. (12:39:31 PM) jsanda: right (12:39:57 PM) jsanda: jshaughn: +1 to more intelligent clients/feeds (12:40:15 PM) jshaughn: Just servers, apps, stiff like that. (12:40:29 PM) jshaughn: ok, I need food... good discussion (2:18:04 PM) pilhuhn: [18:31:43] <mazz> if its faster to just take the data and store it rather than do some pre-filtering to see if we need to insert it or not (2:18:17 PM) pilhuhn: What exactly is slow in getting a value of a local map? (2:18:40 PM) pilhuhn: no, no avail in pinger (well, indirectly via status code) (2:18:55 PM) pilhuhn: And we also did not talk about it in the way of MVP (2:19:36 PM) mazz: (02:18:05 PM) pilhuhn: [18:31:43] <mazz> if its faster to just take the data and store it rather than do some pre-filtering to see if we need to insert it or not (2:20:11 PM) mazz: what I meant was - today in RHQ we get raw avail and before it is inserted, we grab the latest avail data, and if it's the same state, we just update the "end time" (2:20:20 PM) pilhuhn: yes, but that is broken (2:20:24 PM) mazz: if the state changed, we know we have to update the end time and insert a new ros (2:20:25 PM) mazz: row (2:20:34 PM) jshaughn: pilhuhn: I don't think there is anything slow about a map access. But are we sure we'd have a usable cluster-wide map? And if so, who would own it, Hawkular or Metrics? (2:20:40 PM) mazz: so that's the "pre-filtering" I was talking about (2:20:54 PM) pilhuhn: it does not imply though that we just now store everything just because we can not come up with a better solution than in RHQ (2:21:21 PM) mazz: I think we are trying to come up with a better solution :} (2:21:36 PM) pilhuhn: But storing everything is not better (2:22:04 PM) pilhuhn: jshaughn ispn exists .. (2:22:42 PM) pilhuhn: wrt owner -- Hawkular can own it and can own avail (or have a separate component) that "just" uses C* for storage (2:22:48 PM) jshaughn: My feeling is that we should probably store all avail data points for some period of time, and then aggregate to intervals. That gives us our usual high granularity for near term and lowere granularity for longer term (2:23:29 PM) pilhuhn: What is the benefit of stroing 60 times "is up" for an hour as opposed to "was up that hour" ? (2:24:32 PM) jshaughn: ask gaYak and tsegismont_afk, they in particular are thinking people may want to see the more granular data points. (2:24:44 PM) pilhuhn: We can simulate them (2:25:29 PM) pilhuhn: If I know a resources was up last hour, I can return 36000000 individual <time, UP> tuples so that they feel cozy (2:25:37 PM) pilhuhn: Does not even consume storage :-> (2:26:04 PM) pilhuhn: I already see the advertisement "Hawkular has millisecond availability precision" (2:26:06 PM) pilhuhn: :-) (2:26:17 PM) pilhuhn: Sorry for being sarcastic (2:26:20 PM) jshaughn: it;s a bit different than the actual proof that we confirmed UP or DOWN at certain points, but in general I agree that intervals are sufficient (2:26:22 PM) ***pilhuhn should go back to the IDE (2:27:18 PM) jshaughn: there is one subtle thing wrt Alerting. (2:27:20 PM) jsanda: and we can still use/compute/store intervals even if we do initially store everything (2:28:41 PM) jshaughn: it may be better to alert on raw avail than on change-only. I'm not sure. But we saw many times that users tried to use dampening on changes-only and got very confused. (2:28:43 PM) mazz: here was tsegismont_afk use-case, which was a good point: (2:28:43 PM) mazz: (12:14:06 PM) tsegismont: Let's say you have an avail report looking like (2:28:43 PM) mazz: (12:14:17 PM) tsegismont: 5:43:12 UP (2:28:43 PM) mazz: (12:14:32 PM) tsegismont: 5:43:32 UP (2:28:43 PM) mazz: (12:14:41 PM) tsegismont: 5:43:35 DOWN (2:28:43 PM) mazz: (12:15:44 PM) tsegismont: Then you're able to say: it went DOWN at 5:43:35, and last up report was 3 seconds before (2:28:43 PM) mazz: (12:15:52 PM) tsegismont: now if you store only differences (2:28:44 PM) mazz: (12:16:18 PM) tsegismont: and have a report like: (2:28:44 PM) mazz: (12:16:34 PM) tsegismont: 3:43:12 UP (2:28:45 PM) mazz: (12:16:34 PM) tsegismont: 5:43:35 DOWN (2:28:45 PM) mazz: (12:17:03 PM) tsegismont: You can only say: it went DOWN at 5:43:35, and was up since 3:43:12 (2:28:46 PM) mazz: (12:17:52 PM) tsegismont: To make a long story short: if you store things like this, you lose information (2:29:57 PM) gaYak: I tried to say the same thing on the ML ;) (2:30:11 PM) jshaughn: yes, I gave you credit gaYak ;) (2:30:12 PM) pilhuhn: What about then updating the end time on each UP and start a new interval on DOWN? (2:30:33 PM) pilhuhn: Even with above data, we can no tell what happened between ::32 and ::35 (2:30:48 PM) gaYak: No, but the last UP should be stored (2:30:59 PM) gaYak: As in, "it was shown to be up at time X" (2:31:24 PM) gaYak: I'm not sure if the middle points are necessary, but those around the changes (2:31:49 PM) pilhuhn: This is what I meant above - you have a triple <start, end , UP> where start stays as :12 and end is updated on each report , so last at :32> (2:31:52 PM) jshaughn: what gaYak is saying is that we can prove that it was down no more than a certain amount (2:32:26 PM) pilhuhn: I understand that. (2:32:47 PM) pilhuhn: But that still does not warrant storing every single incoming availabilty point (2:33:53 PM) gaYak: Yeah, the issue is that you don't actually know if it's going to be the last 'up' (2:34:00 PM) gaYak: So you need to always store the point when it arrives (2:34:13 PM) jshaughn: If the avail reporting interval is known then the interum points can pretty much be assumed. (2:34:14 PM) gaYak: Sure, you can delete the older, but.. that's even worse for I/O (2:34:40 PM) gaYak: jshaughn: But as our point was to use push towards rhq-metrics.. we might not know (2:35:14 PM) jsanda: how can we guarantee the reporting interval? (2:36:10 PM) gaYak: I don't think we can. Imagine if there's delayed reporting (like a mobile phone which might be out of connection for some time) (2:36:15 PM) jshaughn: but pilhuhn yes, something like UP|startTime|lastTime (2:37:05 PM) pilhuhn: we can't guarantee the reporting interval - never (2:37:13 PM) pilhuhn: a) rest-feeds can do what they want (2:37:25 PM) pilhuhn: b) classical agent can have a hicup that delays it (2:37:45 PM) jsanda: right, so we can't assume data points at particular times (2:37:48 PM) pilhuhn: no (2:38:00 PM) jshaughn: if the cache held the last time that may be sufficient, only writing the lastTime on a change (2:38:35 PM) gaYak: We'll lose cache if the thing goes down, so it needs to be written to Cass every time. (2:38:44 PM) gaYak: And there update equals insert.. (2:38:50 PM) jshaughn: not sure it matters (2:39:12 PM) jshaughn: if we lose lastTime on down (2:39:27 PM) jshaughn: I mean on a crash (2:39:42 PM) jshaughn: may be an acceptable loss (2:39:53 PM) gaYak: Isn't that usually the point when you really need it? ;) (2:40:08 PM) jshaughn: not a hawkular crash, no (2:41:02 PM) gaYak: Cascading failure isn't that uncommon.. only needs one SAN to start behaving badly (2:41:43 PM) gaYak: Also, if we have HA, then we must have our caches synced among every copy of hawkular? (2:41:54 PM) gaYak: Otherwise the lastTime is requestable from only one node? (2:42:45 PM) jshaughn: I think the trade-off of losing that timestamp ona hawkular failure as opposed to millions of unnecessary writes is likely worth it. Not to mention those feeds may be up and running and the cache will pick up where it left off on startup (2:43:23 PM) jshaughn: that was an early question, regarding the nature of the cluster-wide (ispn) cache (2:44:40 PM) jshaughn: but if a feed is likely feeding to the same server repeatedly then the cache sync is maybe less of an issue (2:44:58 PM) jshaughn: i honestly don't know what ispn does across a clustered cache (2:45:33 PM) pilhuhn: There are tons of replication strategies including fully transactional (2:45:38 PM) jsanda: caches can be configured as fully replicated or distributed (2:46:38 PM) jsanda: and if we are going to start talking about using a distributed cache, then we really need to include spark in the discussion as it fits a number of use cases above and beyond being a distributed cache (2:47:22 PM) gaYak: jsanda: Most likely we can't avoid Spark or some other alternative in the long run (2:47:30 PM) gaYak: It seems like such use-cases come up all the time ;) (2:47:39 PM) jsanda: right (2:47:50 PM) jsanda: it's not an all or nothing proposition (2:48:16 PM) jsanda: i alluded to this earlier on the mailing list (2:48:58 PM) jsanda: things like spark, storm, etc. are great fits for a number of different things but the cost of the added operational complexity may very well outweigh the benefits (2:51:38 PM) jshaughn: OK, so where are we? It sounds so far like H Metrics just stores all avail data it receives, and leaves any filtering to a consuming app. It does not need to aggregate unless it wants to (or is potentially configured to do so). For Hawkular it would be unnecessary. Hawkular MVP will assume raw avail as well, for storage and alerting (if it even gets any). Moving forward Hawkular may inrtoduce cache/stream to limit the avail data written to metrics and possibly processed by alerts. Yes? (2:52:15 PM) gaYak: jshaughn: Yeah, probably for MVP it won't matter if we store everything. (2:53:27 PM) jshaughn: H metrics will still need to provide a service to return avail intervals for a requested period, for gui consumption. (2:53:47 PM) jsanda: assuming filtering is what we typically want, i wouldn't have the consuming app do it. have metrics or whatever the service is do it (2:54:49 PM) gaYak: jsanda: mm, I'm not sure if we want to filter it (2:54:53 PM) gaYak: jsanda: From end-applications (2:55:14 PM) gaYak: What if someone wants an alert "monitoring returns something every 1s, otherwise alert" (2:55:19 PM) jshaughn: jsanda: it depends how data flows, for example, if metrics does it then alerts, for example, may not see that benefit (2:55:46 PM) gaYak: jsanda: We might filter on our storage-gateway, but not necessarily from consuming apps (unless they only want availability changes) (2:55:58 PM) gaYak: Which I think should be two different streams.. (2:56:18 PM) gaYak: (yeah, I know getting information about change might be sometimes impossible) (2:56:27 PM) jsanda: i'm lost :) (2:56:40 PM) jsanda: all i'm suggesting is (2:56:50 PM) gaYak: jsanda: If you filter "unnecessary" information on metrics side, while alerting would need it, then that's going to be an issue (2:57:10 PM) jsanda: the query operations provided by the avail service should provide an option to filter out dups (2:57:16 PM) gaYak: Aah (2:57:21 PM) gaYak: Right, we were talking about different things On 2/17/2015 5:52 AM, Heiko W.Rupp wrote:

It make sense to focus on what we need now especially since requirements are still being flushed out. With that said, the first and maybe easiest way I can think of to avoid storing redundant data is to implement a read-before-write pattern which is common and easy enough with the RDBMS. The problem is that read before write does not scale, and I think scalability should be a primary consideration.

Hi, I think we may ned to look at two things - determining availability - storing/displaying it. In RHQ/Jon, the plugin is doing periodic checks and reporting results. If the agent/server connection is down we at some point in time recorded that and did "backfill". RHQ does not allow a resource to say "Hey, I am going down". Nor does it allow to have e.g. a 2nd agent to also check availability by e.g. pinging the resource. Another drawback in RHQ are the sometimes very large reporting intervals that trigger the uncertainty that you mention In Hawkular we need to change the logic in a way that allows for multiple sources to contribute to the overall picture (see for example diagram on https://developer.jboss.org/thread/251479 ). And with those much better information and smaller check intervals, the uncertainty decreases. The whole of availability recording and alerting on it only makes sense if our system notifies the admin before the customer does. I am still convinced that some sort of run length encoding for the stored data < start time, end time (?), state > is the way to go. Especially when the system is not going up and down every other second, but stable for long times. In RHQ we made the error to have a triple < start, end, state> where end was null for an ongoing interval, so we had to not only insert a new <start, end, state> triple, but to also update the previous one with the end time. This is something we should do better this time. Question of getting the time range still remains as even with only tuples of <start, state> for questions like "what was the state last midnight" or "how was availability at Dec 15th at noon". Or for computations of "give me percentage of all availabilty states in the month of January". Here having many many points in time with state makes it easier at the expense of storage usage.