2:39 a.m.
Hey,
we have labels in Hawkular-metrics right now, but apparently there are
use cases that are not yet covered (and I know Matt has more)
- Listing keys of tags. Currently one has to know the available keys to
be able to list the available values
- tag values are currently a comma-separated string and not an array,
which may have implications on allowed characters and escaping of
separators
- Post-tagging of data points (tagging could be provoked by another
system that e.g. parses log files and sees an anomaly)
The idea behind it is that one should be able to create tags at
certain points in time for a single metric or a list of metrics to
uniquely identify that point in time for queries (relative performance
of two versions of a deployment).
What else?
3:29 a.m.
+1 for listing keys when we don't know the metric ids: as you say we need
to know them, or to fetch all metrics definitions, extract and aggregate
tag names; so it's not impossible but a more user friendly method would be
appreciated.
On Mon, Oct 31, 2016 at 8:39 AM, Heiko W.Rupp <hrupp(a)redhat.com> wrote:
Hey,
we have labels in Hawkular-metrics right now, but apparently there are
use cases that are not yet covered (and I know Matt has more)
- Listing keys of tags. Currently one has to know the available keys
to be able to list the available values
- tag values are currently a comma-separated string and not an array,
which may have implications on allowed characters and escaping of separators
- Post-tagging of data points (tagging could be provoked by another
system that e.g. parses log files and sees an anomaly) The idea behind it
is that one should be able to create tags at certain points in time for a
single metric or a list of metrics to uniquely identify that point in time
for queries (relative performance of two versions of a deployment).
What else?
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
9:22 a.m.
On Oct 31, 2016, at 3:39 AM, Heiko W.Rupp <hrupp(a)redhat.com>
wrote:
Hey,
we have labels in Hawkular-metrics right now, but apparently there are
use cases that are not yet covered (and I know Matt has more)
Listing keys of tags. Currently one has to know the available keys to be able to list the
available values
tag values are currently a comma-separated string and not an array, which may have
implications on allowed characters and escaping of separators
Post-tagging of data points (tagging could be provoked by another system that e.g. parses
log files and sees an anomaly) The idea behind it is that one should be able to create
tags at certain points in time for a single metric or a list of metrics to uniquely
identify that point in time for queries (relative performance of two versions of a
deployment).
Data points are immutable by design. Once a row is written it is never
updated. This allows for optimizations with compaction and deletions. We need to take that
into consideration with post-tagging. Using a different table or tables might be a better
option.
4:59 p.m.
----- Original Message -----
From: "John Sanda" <jsanda(a)redhat.com>
To: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Sent: Monday, 31 October, 2016 10:22:19 AM
Subject: Re: [Hawkular-dev] Labeling needs ?
On Oct 31, 2016, at 3:39 AM, Heiko W.Rupp < hrupp(a)redhat.com > wrote:
Hey,
we have labels in Hawkular-metrics right now, but apparently there are
use cases that are not yet covered (and I know Matt has more)
* Listing keys of tags. Currently one has to know the available keys to
be able to list the available values
hmm, interesting, we can list all the values values of a key, but not get the list of the
key itself?
* tag values are currently a comma-separated string and not an
array,
which may have implications on allowed characters and escaping of
separators
The problem with OpenShift is that we have hierarchical data that we would like to store
as tags so that we can perform queries on them.
Most of our data that we store as tags are simple key:value pairs, where the value is a
string. This works nicely with how tags work.
But the more important tag we want to keep track of is labels, and in this case the value
is an array of key:value pairs, which doesn't work too nicely. We store this as a
comma separated string and its possible to use regex in this case, but it starts to be
really messy to write and is prone to errors. Its really not a great solution and looks
really bad when people have to write these complex queries which should be really easy.
* Post-tagging of data points (tagging could be provoked by
another
system that e.g. parses log files and sees an anomaly) The idea behind
it is that one should be able to create tags at certain points in time
for a single metric or a list of metrics to uniquely identify that point
in time for queries (relative performance of two versions of a
deployment).
This feels like an 'event' what could be marked. I don't know if it needs to
be applied to a datapoint directly, or stored in a separate list, or if it even needs to
be stored in Hawkular Metrics.
The main thing I think is to be able to query the list of events which have occurred over
a specific time frame.
OpenShift has a lot of events it can gather, it could be really cool if we could display
these events as marks in the graphs. I don't know if we can use Hawkular Metric string
metrics to store these, or something in Alerts to do it. Heapster has a component to
gather these events for us (https://github.com/kubernetes/heapster/tree/master/events).
Data points are immutable by design. Once a row is written it is
never
updated. This allows for optimizations with compaction and deletions. We
need to take that into consideration with post-tagging. Using a different
table or tables might be a better option.
Do individual data points have tags applied to them? Or only the overall metric?
We may want to keep track of historic changes to tags. I know in OpenShift we store values
in tags which can change at any point in time but I don't believe we can query what
the values was at time X.
10:11 p.m.
On Oct 31, 2016, at 5:59 PM, Matt Wringe <mwringe(a)redhat.com>
wrote:
----- Original Message -----
> From: "John Sanda" <jsanda(a)redhat.com>
> To: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
> Sent: Monday, 31 October, 2016 10:22:19 AM
> Subject: Re: [Hawkular-dev] Labeling needs ?
>
>
>
>
>
> On Oct 31, 2016, at 3:39 AM, Heiko W.Rupp < hrupp(a)redhat.com > wrote:
>
>
>
> Hey,
>
> we have labels in Hawkular-metrics right now, but apparently there are
> use cases that are not yet covered (and I know Matt has more)
>
> * Listing keys of tags. Currently one has to know the available keys to
> be able to list the available values
hmm, interesting, we can list all the values values of a key, but not get the list of the
key itself?
That is correct. We simply have not had a need for it thus far. Adding endpoints to query
tags is straightforward. In fact, it would be a great ticket to work on for anyone looking
to get involved in the project.
> * tag values are currently a comma-separated string and not an array,
> which may have implications on allowed characters and escaping of
> separators
The problem with OpenShift is that we have hierarchical data that we would like to store
as tags so that we can perform queries on them.
Most of our data that we store as tags are simple key:value pairs, where the value is a
string. This works nicely with how tags work.
But the more important tag we want to keep track of is labels, and in this case the value
is an array of key:value pairs, which doesn't work too nicely. We store this as a
comma separated string and its possible to use regex in this case, but it starts to be
really messy to write and is prone to errors. Its really not a great solution and looks
really bad when people have to write these complex queries which should be really easy.
> * Post-tagging of data points (tagging could be provoked by another
> system that e.g. parses log files and sees an anomaly) The idea behind
> it is that one should be able to create tags at certain points in time
> for a single metric or a list of metrics to uniquely identify that point
> in time for queries (relative performance of two versions of a
> deployment).
This feels like an 'event' what could be marked. I don't know if it needs to
be applied to a datapoint directly, or stored in a separate list, or if it even needs to
be stored in Hawkular Metrics.
The main thing I think is to be able to query the list of events which have occurred over
a specific time frame.
OpenShift has a lot of events it can gather, it could be really cool if we could display
these events as marks in the graphs. I don't know if we can use Hawkular Metric string
metrics to store these, or something in Alerts to do it. Heapster has a component to
gather these events for us (https://github.com/kubernetes/heapster/tree/master/events
<https://github.com/kubernetes/heapster/tree/master/events>).
> Data points are immutable by design. Once a row is written it is never
> updated. This allows for optimizations with compaction and deletions. We
> need to take that into consideration with post-tagging. Using a different
> table or tables might be a better option.
Do individual data points have tags applied to them? Or only the overall metric?
We may want to keep track of historic changes to tags. I know in OpenShift we store
values in tags which can change at any point in time but I don't believe we can query
what the values was at time X.
When the oVirt team was looking at Hawkular Metrics, they too were asking for a history of
tag changes. Would this be a generally useful feature?
2:08 a.m.
On 10/31/2016 11:59 PM, Matt Wringe wrote:
hmm, interesting, we can list all the values values of a key, but not
get the list of the key itself?
No one has asked for such feature (or created a
ticket) ;)
Very simple to do though:
return (SELECT DISTINCT tenant_id, tname FROM metrics_tags_idx)
.filter(r -> tenantId.equals(row.getString(0))
.map(r -> r.getString(1))
.distinct()
.toList()
.map(HashSet<T>::new); // Technically redundant, distinct uses a Set already
And there you have it. Now go code ;)
But the more important tag we want to keep track of is labels, and in
this case the value is an array of key:value pairs, which doesn't work
too nicely. We store this as a comma separated string and its possible
to use regex in this case, but it starts to be really messy to write
and is prone to errors. Its really not a great solution and looks
really bad when people have to write these complex queries which
should be really easy.
For quick fix, we could use a workaround like some
applications that
have had this feature introduced later in the life-cycle: Use some
non-visible Unicode to separate the labels. Then just recreate them as
an array when returning the values.
This is simple to do and regexp matching is easy to fix without any user
visibility (we split the value before doing the matching).
This feels like an 'event' what could be marked. I don't
know if it
needs to be applied to a datapoint directly, or stored in a separate
list, or if it even needs to be stored in Hawkular Metrics.
I'd store it separately and then just in the reading phase enrich the
data with this. Such as event "deployment started", "deployment ended"
in the strings metrics (or something like it) and then merge that with
the datapoints fetched.
The main thing I think is to be able to query the list of events
which have occurred over a specific time frame.
And the previous suggestion answers
this one.
OpenShift has a lot of events it can gather, it could be really cool
if we could display these events as marks in the graphs. I don't know if we can use
Hawkular Metric string metrics to store these, or something in Alerts to do it. Heapster
has a component to gather these events for us
(https://github.com/kubernetes/heapster/tree/master/events).
I think we discussed
this year or so ago, but nothing ever happened to
it (I think I even asked if I should implement the events part of
Heapster, but no one needed it at that point).
Actually, I discussed this first time when I sent the original mail to
Google guys if we can create the integration to Hawkular from Heapster.
We may want to keep track of historic changes to tags. I know in
OpenShift we store values in tags which can change at any point in time but I don't
believe we can query what the values was at time X.
This will require some changes
to the data storage, the previous
suggestions do not. This is a snapshot feature, but we don't have that
in Cassandra (unlike Accumulo has for example).
7:23 a.m.
OpenShift has a lot of events it can gather, it could be really cool
if we could display these events as marks in the graphs. I don't know
if we can use Hawkular Metric string metrics to store these, or
something in Alerts to do it. Heapster has a component to gather these
events for us
(https://github.com/kubernetes/heapster/tree/master/events).
It would seem best to consolidate storage in metrics so that data is all
available, and possible more able to be correlated, via the same API.
Havng said that, alerting can store events in its own way. They can be
tagged (no tag change history). They can be queried by date range.
They are partitioned mainly by tenant. The storage model in alerting
may not be as scalable as metrics. Incoming events can optionally be
diverted into trigger evaluation, and in that way incoming events could
trigger other events/alerts, or kick off actions.
3:43 a.m.
On 31 Oct 2016, at 22:59, Matt Wringe wrote:
> * Post-tagging of data points (tagging could be provoked by
> another
> system that e.g. parses log files and sees an anomaly) The idea
> behind
> it is that one should be able to create tags at certain points in
> time
> for a single metric or a list of metrics to uniquely identify
> that point
> in time for queries (relative performance of two versions of a
> deployment).
This feels like an 'event' what could be marked. I don't know if it
needs to be applied to a datapoint directly, or stored in a separate
list, or if it even needs to be stored in Hawkular Metrics.
While I formulated it as it has to be on a (set of) datapoints,
this does not imply any technical implementation.
You are absolutely right, that this could be a separate
map < label, < timestamp, set <metrics>>>
to record the time it applies and the affected metrics.
The main thing I think is to be able to query the list of events
which
have occurred over a specific time frame.
Exactly. And to be able to get metric data in a from-label
to-label (or 'til-now' or '+8h') way
OpenShift has a lot of events it can gather, it could be really cool
if we could display these events as marks in the graphs. I don't know
if we can use Hawkular Metric string
+1
Do individual data points have tags applied to them? Or only the
overall metric?
There are tags on the metric definitions, but I would not want
to overload them with those "events".
We may want to keep track of historic changes to tags. I know in
OpenShift we store values in tags which can change at any point in
time but I don't believe we can query what the values was at time X.
4:30 p.m.
----- Original Message -----
From: "Heiko W.Rupp" <hrupp(a)redhat.com>
To: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Sent: Monday, 31 October, 2016 3:39:46 AM
Subject: [Hawkular-dev] Labeling needs ?
Hey,
we have labels in Hawkular-metrics right now, but apparently there are
use cases that are not yet covered (and I know Matt has more)
* Listing keys of tags. Currently one has to know the available keys to
be able to list the available values
* tag values are currently a comma-separated string and not an
array,
which may have implications on allowed characters and escaping of
separators
* Post-tagging of data points (tagging could be provoked by another
system that e.g. parses log files and sees an anomaly) The idea behind
it is that one should be able to create tags at certain points in time
for a single metric or a list of metrics to uniquely identify that point
in time for queries (relative performance of two versions of a
deployment).
What else?
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2954
days inactive
2955
days old
9 comments
6 participants
participants (6)
-
Heiko W.Rupp -
Jay Shaughnessy -
Joel Takvorian -
John Sanda -
Matt Wringe -
Michael Burman