<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
There was a long IRC chat on this topic yesterday, for thread
completeness I'm adding it here...<br>
<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(11:53:43 AM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>general
question, for feeds sending various types of MetricData, let's say
NumericData and Availability (the only two I know exist currently),
can the data be mixed in a single push, or can we assume that it
will arrive as two different requests, one for each type?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(11:54:15 AM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>stefan_n,
jsanda maybe you have an idea on that already?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#af7f00">(11:55:22 AM) </font></font></span><span
style="font-weight: bold;color: #af7f00;">jsanda: </span>jshaughn:
it's separate<br>
<span style="font-weight: normal;"><font size="2"><font
color="#af7f00">(11:55:47 AM) </font></font></span><span
style="font-weight: bold;color: #af7f00;">jsanda: </span>jshaughn:
but we can certainly change that if it makes sense to do so<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(11:56:56 AM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>jsanda:
no, I think it may be better to have it separate<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(11:57:49 AM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>it
will be easier to parse out the json consistently if it's separate.
also, it may require different processing. Given what we have been
discussing in that avail thread<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(11:58:37 AM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>perhaps
H metrics should (optionally) provide caching of avail data, such
that it gets stored changes-only.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(11:58:58 AM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>not
sure if that should be at the metrics level or higher up<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(11:59:48 AM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>but if
metrics is going to natively handle avail data, then perhaps that
should be a metrics feature. But I'm not sure about the whole
ability of a cluster-wide cache.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:00:01 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>the
other option is the idea of aggregates<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:00:14 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i
would consider it more an implementation detail than a feature<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:00:39 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>I'm
not sure, because users may or may not want changes-only storage<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:01:02 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>your
mention of a cluster-wide cache is timely considering i brought up
spark earlier today on the mailing list<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:01:08 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>then
again, we don't have to be that flexible, we could decide
changes-only is the way it will be<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:01:15 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>users
could get changes only either way<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:01:44 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>right,
the question is whether it's worthwhile to perform the storage of
all the data points<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:02:28 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>when
99% of the avail data is UP, it seems kind of useless to store every
DP<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:02:44 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>data
would be compressed on disk, which i think would make a substantial
difference in terms of disk foot print<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:02:56 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i
haven't thought enough about it either way<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:03:02 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>it's
true what Micke said, about being able to prove the Up-ness was
actually measured, but I'm not convinced it's necessary<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:04:00 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>jsanda:
not sure it matters footprint wise, but it is potentially more
processing, more fetch time, aggregation time, etc<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:04:09 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>time
that could be spent doing other things<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:04:48 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>only
storing changes introduces its own complexity<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:04:53 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>fetch
and store would be the other option<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:05:05 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i
don'nt think so<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:05:47 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>maybe
if we are storing avail in postgres, but definitely not if it is
being stored in C*<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:06:08 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>I
didn't think so, either, I thought I actually saw you propose that
in the thread<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:06:46 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>it's
also possible to do a combination<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:06:52 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>how
do we model absence of value when storing changes only?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:06:56 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>we
store the changes<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:07:08 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>in
addition to all the raw values<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:07:19 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>which
we could roll up<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:07:54 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>and
purge the raw as aggressively as we want<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:08:42 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>tsegismont:
we don't really, I think. Unless we want to keep a counter of how
many repeats we got in an interval (you lose the timestamps but
could likely assume that in most cases they came in at regular
intervals)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:09:03 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>tsegismont:
one way might be for a job to run on the server side and performs
the check for a value being reported within some window<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:09:26 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>jsanda
which means giving intelligence to metrics<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:09:38 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>what
exactly is the need?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:09:39 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>yes<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:09:40 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>which
shouldn't take care of that, IMHO<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:10:00 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>what
do you mean by intelligence?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:10:15 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>"
a job to run on the server side and performs the check for a value
being reported within some window"<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:10:19 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>we
already have use cases for different jobs that will have to run<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:10:27 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>aggregation
could potentially keep the count of the raw metrics collected during
an interval of unchanged avail<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:10:53 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>metrics
itself does know how often a metric/avail should be reported<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:11:06 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>it
takes timestamped-reports, that's it<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:11:17 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>why
can't it know?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:12:02 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>just
like metrics needs to when data retention settings or when various
rollups need to be computed<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:12:29 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>jsanda,
is it stored in metadata?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:13:34 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>it
isn't stored anywhere because we currently aren't doing anything
with availability<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:13:39 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>but it
could be stored<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:14:04 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>Let's
say you have an avail report looking like<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:14:16 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>5:43:12
UP<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:14:31 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>5:43:32
UP<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:14:39 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>5:43:35
DOWN<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:15:43 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>Then
you're able to say: it went DOWN at 5:43:35, and last up report was
3 seconds before<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:15:50 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>now
if you store only differences<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:16:17 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>and
have a report like:<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:16:32 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>3:43:12
UP<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:16:33 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>5:43:35
DOWN<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:17:01 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>You
can only say: it went DOWN at 5:43:35, and was up since 3:43:12<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:17:51 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>To
make a long story short: if you store things like this, you lose
information<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:18:01 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>but
maybe it's not the intention?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:18:29 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>your
example makes sense<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:18:53 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>and i
agree that there may be value in storing all values<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:19:19 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i
haven't formed an opinion one way or the other though<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:19:26 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>need
to think about it more<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:19:42 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>obviously
pros/cons with each<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:22:28 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>if you
stored 3:43:12 UP (N), 43:35 DOWN (2) You'd know that you had 1 UP
in between. If there was some sort of aggregation of the raw data
this may be feasible. You wouldn't know the exact times, but could
likely assume the reports came in at regular intervals<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:23:17 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>what's
the (N) and (2) mean?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#013656">(12:23:32 PM) </font></font></span><span
style="font-weight: normal;color: #013656;">mtho11: </span>if for
this first MVP we just store the changes I think it will make our
lives easier to focus on other higher value things (not to mention
we could change this in phase 2)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#013656">(12:24:02 PM) </font></font></span><span
style="font-weight: normal;color: #013656;">mtho11: </span>we
wouldnt even have do any aggregations on that data most likely<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:24:13 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>mazz,
that would be the count of raw avails contributing the the rollup<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:24:35 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>probably
a dumb idea, but an idea nonetheless<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:25:01 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i
don't think storing only changes is necessarily easier<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:25:08 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>jsanda,
+1<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:25:10 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>in
fact, i think it is harder<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:25:12 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>a lot<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:25:14 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>so if
you have 10000 UPs in a rown before it go's DOWN you'd have DOWN
time (10000)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:25:35 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>because
we need to introduces a caching/clustering layer, i.e., spark<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:25:42 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>storing
changes-only is certainly harder.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:25:44 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>As it
stands right now I think we have to just go with storing all values,
leaving it up to the consuming apps to do anything other than pass
it through in raw form.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:26:08 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>and i
also think it is fair to say it is an implementation detail<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:26:21 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span><jsanda>
in fact, i think it is harder +1<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:26:26 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>right,
the reason why today RHQ does it like it does it (changes only) is
because we would have clobbered the DB with all those avail rows<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:26:38 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>with all
basically the same UP value :)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:27:20 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>but if
C* has better performance for that type of data, then we don't need
to worry about it<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:28:13 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>I
still don't see it as an implementation detail<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:28:45 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i
think the issue won't be so much about write performance as much
about the disk foot print and the cpu needed to aggregate and purge
the raw values<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:28:50 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>because
queries will return different data given the different approaches.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#013656">(12:29:02 PM) </font></font></span><span
style="font-weight: normal;color: #013656;">mtho11: </span>all
long as you can provide the UI with just the changes that is all I
care about :)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:29:10 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>with
that said, there is additional overhead as well for implementing a
solution to only store changes<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:29:21 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>mtho11:
right, that is why it's not an impl detail, I think<br>
<span style="font-weight: normal;"><font size="2"><font
color="#013656">(12:29:37 PM) </font></font></span><span
style="font-weight: normal;color: #013656;">mtho11: </span>great<br>
<span style="font-weight: normal;"><font size="2"><font
color="#af7f00">(12:29:51 PM) </font></font></span><span
style="font-weight: bold;color: #af7f00;">jsanda: </span>jshaughn:
why would the queries return different data?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:30:11 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>if we
have all the values, we can return only the changes<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:30:14 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>right?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:30:18 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>if
it's changes only raw data would return repeats (and likely a lot of
them), if not, it wouldn't.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:30:39 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>so don't
return all the raw data<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:30:43 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>right<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:30:49 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>we
filter the repeats out<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:30:51 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>then
why store it?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:31:02 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>do the
aggregation at query time<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:31:02 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>the
storage is an impl detail because under the covers, we just filter
out what the API returns<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:31:08 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>yep<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:31:10 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>easy
peasy<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:31:13 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>why
store it?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:31:19 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>perhaps
because it is faster?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:31:22 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>I don't
know<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:31:25 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>that
would be a reason though<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:31:33 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>the
argument above was that the details, the repeated values, were
valuable<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:31:34 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>because
it is easier than only storing changes<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(12:31:42 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>if its
faster to just take the data and store it rather than do some
pre-filtering to see if we need to insert it or not<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:31:51 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>and
the example that tsegismont provided is a good one<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:31:58 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>I
think Metrcs would need to have services to return raw and
aggregated<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:33:05 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>we
have a stream of values like<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:33:14 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>tsegismont:
and gaYak were saying the same thing. The question is whether it's
worth the extra storage work. But it's moot if we don't have a good
idea of how to distill raw avail to avail ranges<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:33:30 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>on the
way in<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:33:50 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>that
extra storage work could be less work than only storing the changes<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:33:59 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>personally,
I'm OK with storing it all, at least to start with<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:34:22 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>maybe
it won't be, but i think it will be more work, more resources, etc<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:34:35 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>well,
if we had a global cache it wouldn't be hard to filter duplicates<br>
<span style="font-weight: normal;"><font size="2"><font
color="#4b675b">(12:34:51 PM) </font></font></span><span
style="font-weight: normal;color: #4b675b;">tsegismont: </span>We
don't need to keep that level of details for years<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:35:01 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>right<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:35:17 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>as i
said, i think we could purge the data pretty aggressively<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:35:32 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>so
what is the Spark angle?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:36:14 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>distributed
processing/caching that is fault tolerant and persistent<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:36:45 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>this
is a perfect use case<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:37:26 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i'm
not sure that the benefits outweigh the added complexity though<br>
<span style="font-weight: normal;"><font size="2"><font
color="#013656">(12:37:28 PM) </font></font></span><span
style="font-weight: normal;color: #013656;">mtho11: </span>jsanda:
doesnt purging the data aggressively cause Garbage Collection
issues?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#013656">(12:37:54 PM) </font></font></span><span
style="font-weight: normal;color: #013656;">mtho11: </span>availability
could be on a massive scale<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:38:15 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>not
that i'm aware of<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:39:08 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>we
could consider using date tiered compaction which can drop an entire
sstable at a time<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:39:24 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>so the
purging would be more coarse grained<br>
<span style="font-weight: normal;"><font size="2"><font
color="#013656">(12:39:25 PM) </font></font></span><span
style="font-weight: normal;color: #013656;">mtho11: </span>more
records get purged than get saved<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:39:29 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>one
side-note, I'm hoping that actually a lot of avail filtering is done
by the feeds. Meaning, I think they should report avail only on
critical resources, not *all* of them like in RHQ. That was pretty
useless if you ask me.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(12:39:31 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>right<br>
<span style="font-weight: normal;"><font size="2"><font
color="#af7f00">(12:39:57 PM) </font></font></span><span
style="font-weight: bold;color: #af7f00;">jsanda: </span>jshaughn:
+1 to more intelligent clients/feeds<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:40:15 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>Just
servers, apps, stiff like that.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(12:40:29 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>ok, I
need food... <span style="font-weight: bolder;background: #22ff00;">good
dis</span>cussion<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614"><br>
(2:18:04 PM) </font></font></span><span style="font-weight:
normal;color: #711614;">pilhuhn: </span>[18:31:43] <mazz>
if its faster to just take the data and store it rather than do some
pre-filtering to see if we need to insert it or not<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:18:17 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>What
exactly is slow in getting a value of a local map?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:18:40 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>no,
no avail in pinger (well, indirectly via status code)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:18:55 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>And
we also did not talk about it in the way of MVP<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:19:36 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(02:18:05
PM) pilhuhn: [18:31:43] <mazz> if its faster to just take the
data and store it rather than do some pre-filtering to see if we
need to insert it or not<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:20:11 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>what I
meant was - today in RHQ we get raw avail and before it is inserted,
we grab the latest avail data, and if it's the same state, we just
update the "end time"<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:20:20 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>yes,
but that is broken<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:20:24 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>if the
state changed, we know we have to update the end time and insert a
new ros<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:20:25 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>row<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:20:34 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>pilhuhn:
I don't think there is anything slow about a map access. But are we
sure we'd have a usable cluster-wide map? And if so, who would own
it, Hawkular or Metrics?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:20:40 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>so
that's the "pre-filtering" I was talking about<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:20:54 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>it
does not imply though that we just now store everything just because
we can not come up with a better solution than in RHQ<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:21:21 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>I think
we are trying to come up with a better solution :}<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:21:36 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>But
storing everything is not better<br>
<span style="font-weight: normal;"><font size="2"><font
color="#af7f00">(2:22:04 PM) </font></font></span><span
style="font-weight: bold;color: #af7f00;">pilhuhn: </span>jshaughn
ispn exists ..<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:22:42 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>wrt
owner -- Hawkular can own it and can own avail (or have a separate
component) that "just" uses C* for storage<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:22:48 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>My
feeling is that we should probably store all avail data points for
some period of time, and then aggregate to intervals. That gives us
our usual high granularity for near term and lowere granularity for
longer term<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:23:29 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>What
is the benefit of stroing 60 times "is up" for an hour as opposed to
"was up that hour" ?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:24:32 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>ask
gaYak and tsegismont_afk, they in particular are thinking people may
want to see the more granular data points.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:24:44 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>We
can simulate them<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:25:29 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>If I
know a resources was up last hour, I can return 36000000 individual
<time, UP> tuples so that they feel cozy<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:25:37 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>Does
not even consume storage :-><br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:26:04 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>I
already see the advertisement "Hawkular has millisecond availability
precision"<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:26:06 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>:-)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:26:17 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>Sorry
for being sarcastic<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:26:20 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>it;s a
bit different than the actual proof that we confirmed UP or DOWN at
certain points, but in general I agree that intervals are sufficient<br>
<span style="font-weight: normal;"><font size="2"><font
color="#062585">(2:26:22 PM) </font></font></span><span
style="font-weight: bold;color: #062585;">***pilhuhn </span>should
go back to the IDE<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:27:18 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>there
is one subtle thing wrt Alerting.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:27:20 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>and we
can still use/compute/store intervals even if we do initially store
everything<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:28:41 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>it may
be better to alert on raw avail than on change-only. I'm not sure.
But we saw many times that users tried to use dampening on
changes-only and got very confused.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:43 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>here was
tsegismont_afk use-case, which was a good point:<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:43 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:14:06
PM) tsegismont: Let's say you have an avail report looking like<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:43 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:14:17
PM) tsegismont: 5:43:12 UP<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:43 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:14:32
PM) tsegismont: 5:43:32 UP<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:43 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:14:41
PM) tsegismont: 5:43:35 DOWN<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:43 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:15:44
PM) tsegismont: Then you're able to say: it went DOWN at 5:43:35,
and last up report was 3 seconds before<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:43 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:15:52
PM) tsegismont: now if you store only differences<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:44 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:16:18
PM) tsegismont: and have a report like:<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:44 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:16:34
PM) tsegismont: 3:43:12 UP<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:45 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:16:34
PM) tsegismont: 5:43:35 DOWN<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:45 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:17:03
PM) tsegismont: You can only say: it went DOWN at 5:43:35, and was
up since 3:43:12<br>
<span style="font-weight: normal;"><font size="2"><font
color="#25047c">(2:28:46 PM) </font></font></span><span
style="font-weight: normal;color: #25047c;">mazz: </span>(12:17:52
PM) tsegismont: To make a long story short: if you store things like
this, you lose information<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:29:57 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>I tried
to say the same thing on the ML ;)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:30:11 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>yes, I
gave you credit gaYak ;)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:30:12 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>What
about then updating the end time on each UP and start a new interval
on DOWN?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:30:33 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>Even
with above data, we can no tell what happened between ::32 and ::35<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:30:48 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>No, but
the last UP should be stored<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:30:59 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>As in,
"it was shown to be up at time X"<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:31:24 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>I'm not
sure if the middle points are necessary, but those around the
changes<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:31:49 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>This
is what I meant above - you have a triple <start, end , UP>
where start stays as :12 and end is updated on each report , so last
at :32><br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:31:52 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>what
gaYak is saying is that we can prove that it was down no more than a
certain amount<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:32:26 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>I
understand that.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:32:47 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>But
that still does not warrant storing every single incoming
availabilty point<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:33:53 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Yeah,
the issue is that you don't actually know if it's going to be the
last 'up'<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:34:00 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>So you
need to always store the point when it arrives<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:34:13 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>If the
avail reporting interval is known then the interum points can pretty
much be assumed.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:34:14 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Sure,
you can delete the older, but.. that's even worse for I/O<br>
<span style="font-weight: normal;"><font size="2"><font
color="#af7f00">(2:34:40 PM) </font></font></span><span
style="font-weight: bold;color: #af7f00;">gaYak: </span>jshaughn:
But as our point was to use push towards rhq-metrics.. we might not
know<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:35:14 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>how
can we guarantee the reporting interval?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:36:10 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>I don't
think we can. Imagine if there's delayed reporting (like a mobile
phone which might be out of connection for some time)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:36:15 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>but
pilhuhn yes, something like UP|startTime|lastTime<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:37:05 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>we
can't guarantee the reporting interval - never<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:37:13 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>a)
rest-feeds can do what they want<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:37:25 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>b)
classical agent can have a hicup that delays it<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:37:45 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>right,
so we can't assume data points at particular times<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:37:48 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>no<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:38:00 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>if the
cache held the last time that may be sufficient, only writing the
lastTime on a change<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:38:35 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>We'll
lose cache if the thing goes down, so it needs to be written to Cass
every time.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:38:44 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>And
there update equals insert..<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:38:50 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>not
sure it matters<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:39:12 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>if we
lose lastTime on down<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:39:27 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>I mean
on a crash<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:39:42 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>may be
an acceptable loss<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:39:53 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Isn't
that usually the point when you really need it? ;)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:40:08 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>not a
hawkular crash, no<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:41:02 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Cascading
failure isn't that uncommon.. only needs one SAN to start behaving
badly<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:41:43 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Also,
if we have HA, then we must have our caches synced among every copy
of hawkular?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:41:54 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Otherwise
the lastTime is requestable from only one node?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:42:45 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>I
think the trade-off of losing that timestamp ona hawkular failure as
opposed to millions of unnecessary writes is likely worth it. Not to
mention those feeds may be up and running and the cache will pick up
where it left off on startup<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:43:23 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>that
was an early question, regarding the nature of the cluster-wide
(ispn) cache<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:44:40 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>but if
a feed is likely feeding to the same server repeatedly then the
cache sync is maybe less of an issue<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:44:58 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>i
honestly don't know what ispn does across a clustered cache<br>
<span style="font-weight: normal;"><font size="2"><font
color="#711614">(2:45:33 PM) </font></font></span><span
style="font-weight: normal;color: #711614;">pilhuhn: </span>There
are tons of replication strategies including fully transactional<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:45:38 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>caches
can be configured as fully replicated or distributed<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:46:38 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>and if
we are going to start talking about using a distributed cache, then
we really need to include spark in the discussion as it fits a
number of use cases above and beyond being a distributed cache<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:47:22 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>jsanda:
Most likely we can't avoid Spark or some other alternative in the
long run<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:47:30 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>It
seems like such use-cases come up all the time ;)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:47:39 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>right<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:47:50 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>it's
not an all or nothing proposition<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:48:16 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i
alluded to this earlier on the mailing list<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:48:58 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>things
like spark, storm, etc. are great fits for a number of different
things but the cost of the added operational complexity may very
well outweigh the benefits<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:51:38 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>OK, so
where are we? It sounds so far like H Metrics just stores all avail
data it receives, and leaves any filtering to a consuming app. It
does not need to aggregate unless it wants to (or is potentially
configured to do so). For Hawkular it would be unnecessary. Hawkular
MVP will assume raw avail as well, for storage and alerting (if it
even gets any). Moving forward Hawkular may inrtoduce cache/stream
to limit the avail data written to metrics and possibly processed by
alerts. Yes?<br>
<span style="font-weight: normal;"><font size="2"><font
color="#af7f00">(2:52:15 PM) </font></font></span><span
style="font-weight: bold;color: #af7f00;">gaYak: </span>jshaughn:
Yeah, probably for MVP it won't matter if we store everything.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:53:27 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>H
metrics will still need to provide a service to return avail
intervals for a requested period, for gui consumption.<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:53:47 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>assuming
filtering is what we typically want, i wouldn't have the consuming
app do it. have metrics or whatever the service is do it<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:54:49 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>jsanda:
mm, I'm not sure if we want to filter it<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:54:53 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>jsanda:
From end-applications<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:55:14 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>What if
someone wants an alert "monitoring returns something every 1s,
otherwise alert"<br>
<span style="font-weight: normal;"><font size="2"><font
color="#204a87">(2:55:19 PM) </font></font></span><span
style="font-weight: bold;color: #204a87;">jshaughn: </span>jsanda:
it depends how data flows, for example, if metrics does it then
alerts, for example, may not see that benefit<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:55:46 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>jsanda:
We might filter on our storage-gateway, but not necessarily from
consuming apps (unless they only want availability changes)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:55:58 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Which I
think should be two different streams..<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:56:18 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>(yeah,
I know getting information about change might be sometimes
impossible)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:56:27 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>i'm
lost :)<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:56:40 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>all
i'm suggesting is<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:56:50 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>jsanda:
If you filter "unnecessary" information on metrics side, while
alerting would need it, then that's going to be an issue<br>
<span style="font-weight: normal;"><font size="2"><font
color="#35221e">(2:57:10 PM) </font></font></span><span
style="font-weight: normal;color: #35221e;">jsanda: </span>the
query operations provided by the avail service should provide an
option to filter out dups<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:57:16 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Aah<br>
<span style="font-weight: normal;"><font size="2"><font
color="#a40000">(2:57:21 PM) </font></font></span><span
style="font-weight: normal;color: #a40000;">gaYak: </span>Right,
we were talking about different things<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 2/17/2015 5:52 AM, Heiko W.Rupp
wrote:<br>
</div>
<blockquote
cite="mid:04806D41-CB9C-48D4-9245-FB4D28F69346@redhat.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">Am 17.02.2015 um 10:21 schrieb Thomas Heute <a class="moz-txt-link-rfc2396E" href="mailto:theute@redhat.com"><theute@redhat.com></a>:
IMHO typically one would be happy with a check every 20s/30s or even a minute if the cost is "low", ie: the cost of the monitoring infrastructure is a small fraction of the infrastructure itself.
</pre>
</blockquote>
<pre wrap="">
We need to consider this at different levels.
An agent may be able to determine process availability every second cheaply. This does not imply
that it needs to forward that information every second to the server. And even if it did, it does not
imply that we need to store a data point for every second.
One thing we did never really fit into classical RHQ is caching.
A Map<Long,Byte> for the last known state of a resource (*) can easily store the last known
availability state for a huge number of resources. This way we would not need to query the
database for each incoming point, but we'd look up the value in the cache and only store
when it differs.
Needs a bit more investigating + modelling to expire entries in the cache that do not get a
data point every now and then and inform other subsystems (alerting) - that is not too heavy to implement.
The agent can do similar caching for its resources and only forwarding when the state changes.
This way we can scale to a lot of checks agent side without overwhelming server and data store.
*) Yes I know we are currently using string IDs in many places. We may reconsider that for resource ids.
_______________________________________________
hawkular-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/hawkular-dev">https://lists.jboss.org/mailman/listinfo/hawkular-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>