[Hawkular-dev] Availability (Service and UI)
Jay Shaughnessy
jshaughn at redhat.com
Thu Feb 19 15:54:00 EST 2015
There was a long IRC chat on this topic yesterday, for thread
completeness I'm adding it here...
(11:53:43 AM) jshaughn: general question, for feeds sending various
types of MetricData, let's say NumericData and Availability (the only
two I know exist currently), can the data be mixed in a single push, or
can we assume that it will arrive as two different requests, one for
each type?
(11:54:15 AM) jshaughn: stefan_n, jsanda maybe you have an idea on that
already?
(11:55:22 AM) jsanda: jshaughn: it's separate
(11:55:47 AM) jsanda: jshaughn: but we can certainly change that if it
makes sense to do so
(11:56:56 AM) jshaughn: jsanda: no, I think it may be better to have it
separate
(11:57:49 AM) jshaughn: it will be easier to parse out the json
consistently if it's separate. also, it may require different
processing. Given what we have been discussing in that avail thread
(11:58:37 AM) jshaughn: perhaps H metrics should (optionally) provide
caching of avail data, such that it gets stored changes-only.
(11:58:58 AM) jshaughn: not sure if that should be at the metrics level
or higher up
(11:59:48 AM) jshaughn: but if metrics is going to natively handle avail
data, then perhaps that should be a metrics feature. But I'm not sure
about the whole ability of a cluster-wide cache.
(12:00:01 PM) jshaughn: the other option is the idea of aggregates
(12:00:14 PM) jsanda: i would consider it more an implementation detail
than a feature
(12:00:39 PM) jshaughn: I'm not sure, because users may or may not want
changes-only storage
(12:01:02 PM) jsanda: your mention of a cluster-wide cache is timely
considering i brought up spark earlier today on the mailing list
(12:01:08 PM) jshaughn: then again, we don't have to be that flexible,
we could decide changes-only is the way it will be
(12:01:15 PM) jsanda: users could get changes only either way
(12:01:44 PM) jshaughn: right, the question is whether it's worthwhile
to perform the storage of all the data points
(12:02:28 PM) jshaughn: when 99% of the avail data is UP, it seems kind
of useless to store every DP
(12:02:44 PM) jsanda: data would be compressed on disk, which i think
would make a substantial difference in terms of disk foot print
(12:02:56 PM) jsanda: i haven't thought enough about it either way
(12:03:02 PM) jshaughn: it's true what Micke said, about being able to
prove the Up-ness was actually measured, but I'm not convinced it's
necessary
(12:04:00 PM) jshaughn: jsanda: not sure it matters footprint wise, but
it is potentially more processing, more fetch time, aggregation time, etc
(12:04:09 PM) jshaughn: time that could be spent doing other things
(12:04:48 PM) jsanda: only storing changes introduces its own complexity
(12:04:53 PM) jshaughn: fetch and store would be the other option
(12:05:05 PM) jsanda: i don'nt think so
(12:05:47 PM) jsanda: maybe if we are storing avail in postgres, but
definitely not if it is being stored in C*
(12:06:08 PM) jshaughn: I didn't think so, either, I thought I actually
saw you propose that in the thread
(12:06:46 PM) jsanda: it's also possible to do a combination
(12:06:52 PM) tsegismont: how do we model absence of value when storing
changes only?
(12:06:56 PM) jsanda: we store the changes
(12:07:08 PM) jsanda: in addition to all the raw values
(12:07:19 PM) jsanda: which we could roll up
(12:07:54 PM) jsanda: and purge the raw as aggressively as we want
(12:08:42 PM) jshaughn: tsegismont: we don't really, I think. Unless we
want to keep a counter of how many repeats we got in an interval (you
lose the timestamps but could likely assume that in most cases they came
in at regular intervals)
(12:09:03 PM) jsanda: tsegismont: one way might be for a job to run on
the server side and performs the check for a value being reported within
some window
(12:09:26 PM) tsegismont: jsanda which means giving intelligence to metrics
(12:09:38 PM) jshaughn: what exactly is the need?
(12:09:39 PM) jsanda: yes
(12:09:40 PM) tsegismont: which shouldn't take care of that, IMHO
(12:10:00 PM) jsanda: what do you mean by intelligence?
(12:10:15 PM) tsegismont: " a job to run on the server side and performs
the check for a value being reported within some window"
(12:10:19 PM) jsanda: we already have use cases for different jobs that
will have to run
(12:10:27 PM) jshaughn: aggregation could potentially keep the count of
the raw metrics collected during an interval of unchanged avail
(12:10:53 PM) tsegismont: metrics itself does know how often a
metric/avail should be reported
(12:11:06 PM) tsegismont: it takes timestamped-reports, that's it
(12:11:17 PM) jsanda: why can't it know?
(12:12:02 PM) jsanda: just like metrics needs to when data retention
settings or when various rollups need to be computed
(12:12:29 PM) tsegismont: jsanda, is it stored in metadata?
(12:13:34 PM) jsanda: it isn't stored anywhere because we currently
aren't doing anything with availability
(12:13:39 PM) jsanda: but it could be stored
(12:14:04 PM) tsegismont: Let's say you have an avail report looking like
(12:14:16 PM) tsegismont: 5:43:12 UP
(12:14:31 PM) tsegismont: 5:43:32 UP
(12:14:39 PM) tsegismont: 5:43:35 DOWN
(12:15:43 PM) tsegismont: Then you're able to say: it went DOWN at
5:43:35, and last up report was 3 seconds before
(12:15:50 PM) tsegismont: now if you store only differences
(12:16:17 PM) tsegismont: and have a report like:
(12:16:32 PM) tsegismont: 3:43:12 UP
(12:16:33 PM) tsegismont: 5:43:35 DOWN
(12:17:01 PM) tsegismont: You can only say: it went DOWN at 5:43:35, and
was up since 3:43:12
(12:17:51 PM) tsegismont: To make a long story short: if you store
things like this, you lose information
(12:18:01 PM) tsegismont: but maybe it's not the intention?
(12:18:29 PM) jsanda: your example makes sense
(12:18:53 PM) jsanda: and i agree that there may be value in storing all
values
(12:19:19 PM) jsanda: i haven't formed an opinion one way or the other
though
(12:19:26 PM) jsanda: need to think about it more
(12:19:42 PM) jsanda: obviously pros/cons with each
(12:22:28 PM) jshaughn: if you stored 3:43:12 UP (N), 43:35 DOWN (2)
You'd know that you had 1 UP in between. If there was some sort of
aggregation of the raw data this may be feasible. You wouldn't know the
exact times, but could likely assume the reports came in at regular
intervals
(12:23:17 PM) mazz: what's the (N) and (2) mean?
(12:23:32 PM) mtho11: if for this first MVP we just store the changes I
think it will make our lives easier to focus on other higher value
things (not to mention we could change this in phase 2)
(12:24:02 PM) mtho11: we wouldnt even have do any aggregations on that
data most likely
(12:24:13 PM) jshaughn: mazz, that would be the count of raw avails
contributing the the rollup
(12:24:35 PM) jshaughn: probably a dumb idea, but an idea nonetheless
(12:25:01 PM) jsanda: i don't think storing only changes is necessarily
easier
(12:25:08 PM) tsegismont: jsanda, +1
(12:25:10 PM) jsanda: in fact, i think it is harder
(12:25:12 PM) jsanda: a lot
(12:25:14 PM) jshaughn: so if you have 10000 UPs in a rown before it
go's DOWN you'd have DOWN time (10000)
(12:25:35 PM) jsanda: because we need to introduces a caching/clustering
layer, i.e., spark
(12:25:42 PM) jshaughn: storing changes-only is certainly harder.
(12:25:44 PM) jshaughn: As it stands right now I think we have to just
go with storing all values, leaving it up to the consuming apps to do
anything other than pass it through in raw form.
(12:26:08 PM) jsanda: and i also think it is fair to say it is an
implementation detail
(12:26:21 PM) tsegismont: <jsanda> in fact, i think it is harder +1
(12:26:26 PM) mazz: right, the reason why today RHQ does it like it does
it (changes only) is because we would have clobbered the DB with all
those avail rows
(12:26:38 PM) mazz: with all basically the same UP value :)
(12:27:20 PM) mazz: but if C* has better performance for that type of
data, then we don't need to worry about it
(12:28:13 PM) jshaughn: I still don't see it as an implementation detail
(12:28:45 PM) jsanda: i think the issue won't be so much about write
performance as much about the disk foot print and the cpu needed to
aggregate and purge the raw values
(12:28:50 PM) jshaughn: because queries will return different data given
the different approaches.
(12:29:02 PM) mtho11: all long as you can provide the UI with just the
changes that is all I care about :)
(12:29:10 PM) jsanda: with that said, there is additional overhead as
well for implementing a solution to only store changes
(12:29:21 PM) jshaughn: mtho11: right, that is why it's not an impl
detail, I think
(12:29:37 PM) mtho11: great
(12:29:51 PM) jsanda: jshaughn: why would the queries return different data?
(12:30:11 PM) jsanda: if we have all the values, we can return only the
changes
(12:30:14 PM) jsanda: right?
(12:30:18 PM) jshaughn: if it's changes only raw data would return
repeats (and likely a lot of them), if not, it wouldn't.
(12:30:39 PM) mazz: so don't return all the raw data
(12:30:43 PM) jsanda: right
(12:30:49 PM) jsanda: we filter the repeats out
(12:30:51 PM) jshaughn: then why store it?
(12:31:02 PM) jsanda: do the aggregation at query time
(12:31:02 PM) mazz: the storage is an impl detail because under the
covers, we just filter out what the API returns
(12:31:08 PM) jsanda: yep
(12:31:10 PM) jsanda: easy peasy
(12:31:13 PM) mazz: why store it?
(12:31:19 PM) mazz: perhaps because it is faster?
(12:31:22 PM) mazz: I don't know
(12:31:25 PM) mazz: that would be a reason though
(12:31:33 PM) jshaughn: the argument above was that the details, the
repeated values, were valuable
(12:31:34 PM) jsanda: because it is easier than only storing changes
(12:31:42 PM) mazz: if its faster to just take the data and store it
rather than do some pre-filtering to see if we need to insert it or not
(12:31:51 PM) jsanda: and the example that tsegismont provided is a good one
(12:31:58 PM) jshaughn: I think Metrcs would need to have services to
return raw and aggregated
(12:33:05 PM) jsanda: we have a stream of values like
(12:33:14 PM) jshaughn: tsegismont: and gaYak were saying the same
thing. The question is whether it's worth the extra storage work. But
it's moot if we don't have a good idea of how to distill raw avail to
avail ranges
(12:33:30 PM) jshaughn: on the way in
(12:33:50 PM) jsanda: that extra storage work could be less work than
only storing the changes
(12:33:59 PM) jshaughn: personally, I'm OK with storing it all, at least
to start with
(12:34:22 PM) jsanda: maybe it won't be, but i think it will be more
work, more resources, etc
(12:34:35 PM) jshaughn: well, if we had a global cache it wouldn't be
hard to filter duplicates
(12:34:51 PM) tsegismont: We don't need to keep that level of details
for years
(12:35:01 PM) jsanda: right
(12:35:17 PM) jsanda: as i said, i think we could purge the data pretty
aggressively
(12:35:32 PM) jshaughn: so what is the Spark angle?
(12:36:14 PM) jsanda: distributed processing/caching that is fault
tolerant and persistent
(12:36:45 PM) jsanda: this is a perfect use case
(12:37:26 PM) jsanda: i'm not sure that the benefits outweigh the added
complexity though
(12:37:28 PM) mtho11: jsanda: doesnt purging the data aggressively cause
Garbage Collection issues?
(12:37:54 PM) mtho11: availability could be on a massive scale
(12:38:15 PM) jsanda: not that i'm aware of
(12:39:08 PM) jsanda: we could consider using date tiered compaction
which can drop an entire sstable at a time
(12:39:24 PM) jsanda: so the purging would be more coarse grained
(12:39:25 PM) mtho11: more records get purged than get saved
(12:39:29 PM) jshaughn: one side-note, I'm hoping that actually a lot of
avail filtering is done by the feeds. Meaning, I think they should
report avail only on critical resources, not *all* of them like in RHQ.
That was pretty useless if you ask me.
(12:39:31 PM) jsanda: right
(12:39:57 PM) jsanda: jshaughn: +1 to more intelligent clients/feeds
(12:40:15 PM) jshaughn: Just servers, apps, stiff like that.
(12:40:29 PM) jshaughn: ok, I need food... good discussion
(2:18:04 PM) pilhuhn: [18:31:43] <mazz> if its faster to just take the
data and store it rather than do some pre-filtering to see if we need to
insert it or not
(2:18:17 PM) pilhuhn: What exactly is slow in getting a value of a local
map?
(2:18:40 PM) pilhuhn: no, no avail in pinger (well, indirectly via
status code)
(2:18:55 PM) pilhuhn: And we also did not talk about it in the way of MVP
(2:19:36 PM) mazz: (02:18:05 PM) pilhuhn: [18:31:43] <mazz> if its
faster to just take the data and store it rather than do some
pre-filtering to see if we need to insert it or not
(2:20:11 PM) mazz: what I meant was - today in RHQ we get raw avail and
before it is inserted, we grab the latest avail data, and if it's the
same state, we just update the "end time"
(2:20:20 PM) pilhuhn: yes, but that is broken
(2:20:24 PM) mazz: if the state changed, we know we have to update the
end time and insert a new ros
(2:20:25 PM) mazz: row
(2:20:34 PM) jshaughn: pilhuhn: I don't think there is anything slow
about a map access. But are we sure we'd have a usable cluster-wide map?
And if so, who would own it, Hawkular or Metrics?
(2:20:40 PM) mazz: so that's the "pre-filtering" I was talking about
(2:20:54 PM) pilhuhn: it does not imply though that we just now store
everything just because we can not come up with a better solution than
in RHQ
(2:21:21 PM) mazz: I think we are trying to come up with a better
solution :}
(2:21:36 PM) pilhuhn: But storing everything is not better
(2:22:04 PM) pilhuhn: jshaughn ispn exists ..
(2:22:42 PM) pilhuhn: wrt owner -- Hawkular can own it and can own avail
(or have a separate component) that "just" uses C* for storage
(2:22:48 PM) jshaughn: My feeling is that we should probably store all
avail data points for some period of time, and then aggregate to
intervals. That gives us our usual high granularity for near term and
lowere granularity for longer term
(2:23:29 PM) pilhuhn: What is the benefit of stroing 60 times "is up"
for an hour as opposed to "was up that hour" ?
(2:24:32 PM) jshaughn: ask gaYak and tsegismont_afk, they in particular
are thinking people may want to see the more granular data points.
(2:24:44 PM) pilhuhn: We can simulate them
(2:25:29 PM) pilhuhn: If I know a resources was up last hour, I can
return 36000000 individual <time, UP> tuples so that they feel cozy
(2:25:37 PM) pilhuhn: Does not even consume storage :->
(2:26:04 PM) pilhuhn: I already see the advertisement "Hawkular has
millisecond availability precision"
(2:26:06 PM) pilhuhn: :-)
(2:26:17 PM) pilhuhn: Sorry for being sarcastic
(2:26:20 PM) jshaughn: it;s a bit different than the actual proof that
we confirmed UP or DOWN at certain points, but in general I agree that
intervals are sufficient
(2:26:22 PM) ***pilhuhn should go back to the IDE
(2:27:18 PM) jshaughn: there is one subtle thing wrt Alerting.
(2:27:20 PM) jsanda: and we can still use/compute/store intervals even
if we do initially store everything
(2:28:41 PM) jshaughn: it may be better to alert on raw avail than on
change-only. I'm not sure. But we saw many times that users tried to use
dampening on changes-only and got very confused.
(2:28:43 PM) mazz: here was tsegismont_afk use-case, which was a good point:
(2:28:43 PM) mazz: (12:14:06 PM) tsegismont: Let's say you have an avail
report looking like
(2:28:43 PM) mazz: (12:14:17 PM) tsegismont: 5:43:12 UP
(2:28:43 PM) mazz: (12:14:32 PM) tsegismont: 5:43:32 UP
(2:28:43 PM) mazz: (12:14:41 PM) tsegismont: 5:43:35 DOWN
(2:28:43 PM) mazz: (12:15:44 PM) tsegismont: Then you're able to say: it
went DOWN at 5:43:35, and last up report was 3 seconds before
(2:28:43 PM) mazz: (12:15:52 PM) tsegismont: now if you store only
differences
(2:28:44 PM) mazz: (12:16:18 PM) tsegismont: and have a report like:
(2:28:44 PM) mazz: (12:16:34 PM) tsegismont: 3:43:12 UP
(2:28:45 PM) mazz: (12:16:34 PM) tsegismont: 5:43:35 DOWN
(2:28:45 PM) mazz: (12:17:03 PM) tsegismont: You can only say: it went
DOWN at 5:43:35, and was up since 3:43:12
(2:28:46 PM) mazz: (12:17:52 PM) tsegismont: To make a long story short:
if you store things like this, you lose information
(2:29:57 PM) gaYak: I tried to say the same thing on the ML ;)
(2:30:11 PM) jshaughn: yes, I gave you credit gaYak ;)
(2:30:12 PM) pilhuhn: What about then updating the end time on each UP
and start a new interval on DOWN?
(2:30:33 PM) pilhuhn: Even with above data, we can no tell what happened
between ::32 and ::35
(2:30:48 PM) gaYak: No, but the last UP should be stored
(2:30:59 PM) gaYak: As in, "it was shown to be up at time X"
(2:31:24 PM) gaYak: I'm not sure if the middle points are necessary, but
those around the changes
(2:31:49 PM) pilhuhn: This is what I meant above - you have a triple
<start, end , UP> where start stays as :12 and end is updated on each
report , so last at :32>
(2:31:52 PM) jshaughn: what gaYak is saying is that we can prove that it
was down no more than a certain amount
(2:32:26 PM) pilhuhn: I understand that.
(2:32:47 PM) pilhuhn: But that still does not warrant storing every
single incoming availabilty point
(2:33:53 PM) gaYak: Yeah, the issue is that you don't actually know if
it's going to be the last 'up'
(2:34:00 PM) gaYak: So you need to always store the point when it arrives
(2:34:13 PM) jshaughn: If the avail reporting interval is known then the
interum points can pretty much be assumed.
(2:34:14 PM) gaYak: Sure, you can delete the older, but.. that's even
worse for I/O
(2:34:40 PM) gaYak: jshaughn: But as our point was to use push towards
rhq-metrics.. we might not know
(2:35:14 PM) jsanda: how can we guarantee the reporting interval?
(2:36:10 PM) gaYak: I don't think we can. Imagine if there's delayed
reporting (like a mobile phone which might be out of connection for some
time)
(2:36:15 PM) jshaughn: but pilhuhn yes, something like UP|startTime|lastTime
(2:37:05 PM) pilhuhn: we can't guarantee the reporting interval - never
(2:37:13 PM) pilhuhn: a) rest-feeds can do what they want
(2:37:25 PM) pilhuhn: b) classical agent can have a hicup that delays it
(2:37:45 PM) jsanda: right, so we can't assume data points at particular
times
(2:37:48 PM) pilhuhn: no
(2:38:00 PM) jshaughn: if the cache held the last time that may be
sufficient, only writing the lastTime on a change
(2:38:35 PM) gaYak: We'll lose cache if the thing goes down, so it needs
to be written to Cass every time.
(2:38:44 PM) gaYak: And there update equals insert..
(2:38:50 PM) jshaughn: not sure it matters
(2:39:12 PM) jshaughn: if we lose lastTime on down
(2:39:27 PM) jshaughn: I mean on a crash
(2:39:42 PM) jshaughn: may be an acceptable loss
(2:39:53 PM) gaYak: Isn't that usually the point when you really need it? ;)
(2:40:08 PM) jshaughn: not a hawkular crash, no
(2:41:02 PM) gaYak: Cascading failure isn't that uncommon.. only needs
one SAN to start behaving badly
(2:41:43 PM) gaYak: Also, if we have HA, then we must have our caches
synced among every copy of hawkular?
(2:41:54 PM) gaYak: Otherwise the lastTime is requestable from only one
node?
(2:42:45 PM) jshaughn: I think the trade-off of losing that timestamp
ona hawkular failure as opposed to millions of unnecessary writes is
likely worth it. Not to mention those feeds may be up and running and
the cache will pick up where it left off on startup
(2:43:23 PM) jshaughn: that was an early question, regarding the nature
of the cluster-wide (ispn) cache
(2:44:40 PM) jshaughn: but if a feed is likely feeding to the same
server repeatedly then the cache sync is maybe less of an issue
(2:44:58 PM) jshaughn: i honestly don't know what ispn does across a
clustered cache
(2:45:33 PM) pilhuhn: There are tons of replication strategies including
fully transactional
(2:45:38 PM) jsanda: caches can be configured as fully replicated or
distributed
(2:46:38 PM) jsanda: and if we are going to start talking about using a
distributed cache, then we really need to include spark in the
discussion as it fits a number of use cases above and beyond being a
distributed cache
(2:47:22 PM) gaYak: jsanda: Most likely we can't avoid Spark or some
other alternative in the long run
(2:47:30 PM) gaYak: It seems like such use-cases come up all the time ;)
(2:47:39 PM) jsanda: right
(2:47:50 PM) jsanda: it's not an all or nothing proposition
(2:48:16 PM) jsanda: i alluded to this earlier on the mailing list
(2:48:58 PM) jsanda: things like spark, storm, etc. are great fits for a
number of different things but the cost of the added operational
complexity may very well outweigh the benefits
(2:51:38 PM) jshaughn: OK, so where are we? It sounds so far like H
Metrics just stores all avail data it receives, and leaves any filtering
to a consuming app. It does not need to aggregate unless it wants to (or
is potentially configured to do so). For Hawkular it would be
unnecessary. Hawkular MVP will assume raw avail as well, for storage and
alerting (if it even gets any). Moving forward Hawkular may inrtoduce
cache/stream to limit the avail data written to metrics and possibly
processed by alerts. Yes?
(2:52:15 PM) gaYak: jshaughn: Yeah, probably for MVP it won't matter if
we store everything.
(2:53:27 PM) jshaughn: H metrics will still need to provide a service to
return avail intervals for a requested period, for gui consumption.
(2:53:47 PM) jsanda: assuming filtering is what we typically want, i
wouldn't have the consuming app do it. have metrics or whatever the
service is do it
(2:54:49 PM) gaYak: jsanda: mm, I'm not sure if we want to filter it
(2:54:53 PM) gaYak: jsanda: From end-applications
(2:55:14 PM) gaYak: What if someone wants an alert "monitoring returns
something every 1s, otherwise alert"
(2:55:19 PM) jshaughn: jsanda: it depends how data flows, for example,
if metrics does it then alerts, for example, may not see that benefit
(2:55:46 PM) gaYak: jsanda: We might filter on our storage-gateway, but
not necessarily from consuming apps (unless they only want availability
changes)
(2:55:58 PM) gaYak: Which I think should be two different streams..
(2:56:18 PM) gaYak: (yeah, I know getting information about change might
be sometimes impossible)
(2:56:27 PM) jsanda: i'm lost :)
(2:56:40 PM) jsanda: all i'm suggesting is
(2:56:50 PM) gaYak: jsanda: If you filter "unnecessary" information on
metrics side, while alerting would need it, then that's going to be an issue
(2:57:10 PM) jsanda: the query operations provided by the avail service
should provide an option to filter out dups
(2:57:16 PM) gaYak: Aah
(2:57:21 PM) gaYak: Right, we were talking about different things
On 2/17/2015 5:52 AM, Heiko W.Rupp wrote:
>> Am 17.02.2015 um 10:21 schrieb Thomas Heute <theute at redhat.com>:
>> IMHO typically one would be happy with a check every 20s/30s or even a minute if the cost is "low", ie: the cost of the monitoring infrastructure is a small fraction of the infrastructure itself.
> We need to consider this at different levels.
>
> An agent may be able to determine process availability every second cheaply. This does not imply
> that it needs to forward that information every second to the server. And even if it did, it does not
> imply that we need to store a data point for every second.
>
> One thing we did never really fit into classical RHQ is caching.
> A Map<Long,Byte> for the last known state of a resource (*) can easily store the last known
> availability state for a huge number of resources. This way we would not need to query the
> database for each incoming point, but we'd look up the value in the cache and only store
> when it differs.
> Needs a bit more investigating + modelling to expire entries in the cache that do not get a
> data point every now and then and inform other subsystems (alerting) - that is not too heavy to implement.
> The agent can do similar caching for its resources and only forwarding when the state changes.
>
> This way we can scale to a lot of checks agent side without overwhelming server and data store.
>
>
> *) Yes I know we are currently using string IDs in many places. We may reconsider that for resource ids.
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20150219/dc1fe779/attachment-0001.html
More information about the hawkular-dev
mailing list