3:13 a.m.
<tl;dr> ======
Agent is introducing two changes:
1. Metric Type definitions created by the agent will have the same ID as before, but its
Name is changing (probably does not affect anyone).
2. Clients (like UI, HawkFX, etc) should no longer assume the agent's h-inventory
metric definition IDs match h-metric metric IDs - instead, they must look at the
"metric-id" property on the h-inventory metric definition to know how to look up
the actual metric data in h-metrics. Will affect all clients, but metric IDs will default
to what they are today - so nothing changes and thus nothing will break today if you run
with out of box configuration.
</tl;dr> ======
There is a use-case where the agent needs to support custom metric IDs (that is, rather
than accepting the out-of-box metric IDs created by the agent, allow the user to define
what the metric IDs should look like). See https://issues.jboss.org/browse/HWKAGENT-78
As a refresher, remember that when you create resources in inventory, those resources can
be associated with one or more "metric" definitions. Those resource metrics are
themselves associated with a "metric type" definition. Today, when the agent
stores metric data into Hawkular Metrics, it stores the data under the ID of the
"metric" that is associated with the resource (so the h-inventory metric ID is
the same as the h-metric metric ID by definition - at least for the data the hawkular
wildfly agent inserts).
I am proposing two changes in the PR: https://github.com/hawkular/hawkular-agent/pull/226
First, today, the "metric type" definition that the agent creates has an ID and
a Name that are identical. I am changing this so the ID stays the same (which is the
metric set name, followed by "~", followed by the name of the metric -- e.g. if
there was a <metric-set-dmr name="this"> that contains a <metric-dmr
name="that">, the metric type ID would be "this~that"), but the
Name is only the name without the set name (e.g. the name would be "that" in the
previous example).
The above is a minor change, and I doubt anyone is affected by it. But I point it out just
in case.
Second, it should no longer be assumed that the inventory's resource metric ID is
identical to the h-metric's metric ID.
This second change will potentially affect everyone (I know it affects Heiko's HawkFX
:)
Now, that said, nothing really changes now, because the defaults will remain as they are
(that is, the h-inventory's metric ID will still be exactly the same as the h-metrics
ID - the agent keeps them identical). The change happens when the user actually configures
the agent with a custom metric ID template (e.g. <remote-jmx
metric-id-template="${pod.id}-%MetricTypeName" ...>). This means h-metric IDs
will be DIFFERENT than h-inventory metric IDs.
How then does a client know what h-metric IDs to look for if they only have h-inventory
metric definitions? Well, recall that inventory allows for properties to be associated
with any entity. I use this feature here. Rather than rely on an implicit rule
("h-inventory metric ID is the same as h-metric metric ID") I explicitly define
this linkage in a property called "metric-id" on the h-inventory metric
definition. Out of box, that property's value will be identical to the h-inventory
metric ID (and hence why nothing really changes - since the explicit rule in this case
provides the same behavior as if following the old implicit rule). In fact, I'm
considering if I should set that property at all if its the same as the h-inventory ID - I
think it might be better to only set a "metric-id" property if it is different.
But this would require clients to know about the implicit rule if there is no metric-id
property set ("is there a metric-id property set? No? Then use the h-inventory metric
ID for the h-metric metric ID").
For example, see here (this is a live example I copied from the "raw" inventory
JSON that HawkFX gave me for a metric) - this is the h-inventory entity definition for the
metric "Heap Used" on my WildFly Server resource - notice the
"properties" map has a "metric-id" value that is DIFFERENT than the
"id" - that "metric-id" is something I customized in my agent config
in standalone.xml (well, I used the swarm agent, so I put it in the swarm config, but its
basically the same thing):
{
"path":
"/t;hawkular/f;mazz/m;MI~R~%5Bmazz%2FWildFly~~%5D~MT~WildFly%20Memory%20Metrics~Heap%20Used",
"properties": {
"__identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"metric-id": "feed=mazz__msn=WildFly__typeName=Heap
Used__resName=WildFly Server [WildFly]__resId=WildFly~~__typeId=WildFly Memory
Metrics~Heap Used"
},
"name": "Heap Used",
"identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"type": {
"path":
"/t;hawkular/f;mazz/mt;WildFly%20Memory%20Metrics~Heap%20Used",
"name": "Heap Used",
"identityHash": "3be5b5fdabed925ac46fdc6d8295e34bbd3147a",
"unit": "BYTES",
"type": "GAUGE",
"collectionInterval": 30,
"id": "WildFly Memory Metrics~Heap Used"
},
"id": "MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used"
}
Notice this:
feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server
[WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used
is different from this:
MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used
And that's the issue. Clients have to know to look for the "metric-id"
property and use it when looking up metric data in h-metrics (so if you want to graph the
data, you have to ask h-metrics for the data associated with the value found in the
"metric-id" property).
8:56 a.m.
Hi John,
Thanks for bringing this as it might have potential side effects.
I haven't reviewed your proposal yet but you can take an idea about how MiQ links
Hawkular metrics inside something that can be managed internally of MiQ.
https://github.com/ManageIQ/manageiq/blob/master/product/live_metrics/mid...
https://github.com/ManageIQ/manageiq/blob/master/product/live_metrics/mid...
I guess that if the id of the type is still valid, then perhaps nothing affects but just
to be sure I chime in here just to validate it.
I am afraid about potential changes at this stage.
----- Mensaje original -----
De: "John Mazzitelli" <mazz(a)redhat.com>
Para: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Enviados: Jueves, 7 de Julio 2016 3:13:37
Asunto: [Hawkular-dev] agent using custom metric IDs
<tl;dr> ======
Agent is introducing two changes:
1. Metric Type definitions created by the agent will have the same ID as
before, but its Name is changing (probably does not affect anyone).
2. Clients (like UI, HawkFX, etc) should no longer assume the agent's
h-inventory metric definition IDs match h-metric metric IDs - instead, they
must look at the "metric-id" property on the h-inventory metric definition
to know how to look up the actual metric data in h-metrics. Will affect all
clients, but metric IDs will default to what they are today - so nothing
changes and thus nothing will break today if you run with out of box
configuration.
</tl;dr> ======
There is a use-case where the agent needs to support custom metric IDs (that
is, rather than accepting the out-of-box metric IDs created by the agent,
allow the user to define what the metric IDs should look like). See
https://issues.jboss.org/browse/HWKAGENT-78
As a refresher, remember that when you create resources in inventory, those
resources can be associated with one or more "metric" definitions. Those
resource metrics are themselves associated with a "metric type" definition.
Today, when the agent stores metric data into Hawkular Metrics, it stores
the data under the ID of the "metric" that is associated with the resource
(so the h-inventory metric ID is the same as the h-metric metric ID by
definition - at least for the data the hawkular wildfly agent inserts).
I am proposing two changes in the PR:
https://github.com/hawkular/hawkular-agent/pull/226
First, today, the "metric type" definition that the agent creates has an ID
and a Name that are identical. I am changing this so the ID stays the same
(which is the metric set name, followed by "~", followed by the name of the
metric -- e.g. if there was a <metric-set-dmr name="this"> that contains
a
<metric-dmr name="that">, the metric type ID would be
"this~that"), but the
Name is only the name without the set name (e.g. the name would be "that" in
the previous example).
The above is a minor change, and I doubt anyone is affected by it. But I
point it out just in case.
Second, it should no longer be assumed that the inventory's resource metric
ID is identical to the h-metric's metric ID.
This second change will potentially affect everyone (I know it affects
Heiko's HawkFX :)
Now, that said, nothing really changes now, because the defaults will remain
as they are (that is, the h-inventory's metric ID will still be exactly the
same as the h-metrics ID - the agent keeps them identical). The change
happens when the user actually configures the agent with a custom metric ID
template (e.g. <remote-jmx metric-id-template="${pod.id}-%MetricTypeName"
...>). This means h-metric IDs will be DIFFERENT than h-inventory metric
IDs.
How then does a client know what h-metric IDs to look for if they only have
h-inventory metric definitions? Well, recall that inventory allows for
properties to be associated with any entity. I use this feature here. Rather
than rely on an implicit rule ("h-inventory metric ID is the same as
h-metric metric ID") I explicitly define this linkage in a property called
"metric-id" on the h-inventory metric definition. Out of box, that
property's value will be identical to the h-inventory metric ID (and hence
why nothing really changes - since the explicit rule in this case provides
the same behavior as if following the old implicit rule). In fact, I'm
considering if I should set that property at all if its the same as the
h-inventory ID - I think it might be better to only set a "metric-id"
property if it is different. But this would require clients to know about
the implicit rule if there is no metric-id property set ("is there a
metric-id property set? No? Then use the h-inven!
tory metric ID for the h-metric metric ID").
For example, see here (this is a live example I copied from the "raw"
inventory JSON that HawkFX gave me for a metric) - this is the h-inventory
entity definition for the metric "Heap Used" on my WildFly Server resource -
notice the "properties" map has a "metric-id" value that is DIFFERENT
than
the "id" - that "metric-id" is something I customized in my agent
config in
standalone.xml (well, I used the swarm agent, so I put it in the swarm
config, but its basically the same thing):
{
"path":
"/t;hawkular/f;mazz/m;MI~R~%5Bmazz%2FWildFly~~%5D~MT~WildFly%20Memory%20Metrics~Heap%20Used",
"properties": {
"__identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"metric-id": "feed=mazz__msn=WildFly__typeName=Heap
Used__resName=WildFly
Server [WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap
Used"
},
"name": "Heap Used",
"identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"type": {
"path":
"/t;hawkular/f;mazz/mt;WildFly%20Memory%20Metrics~Heap%20Used",
"name": "Heap Used",
"identityHash": "3be5b5fdabed925ac46fdc6d8295e34bbd3147a",
"unit": "BYTES",
"type": "GAUGE",
"collectionInterval": 30,
"id": "WildFly Memory Metrics~Heap Used"
},
"id": "MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used"
}
Notice this:
feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server
[WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used
is different from this:
MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used
And that's the issue. Clients have to know to look for the "metric-id"
property and use it when looking up metric data in h-metrics (so if you want
to graph the data, you have to ask h-metrics for the data associated with
the value found in the "metric-id" property).
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2:45 p.m.
I haven't reviewed your proposal yet but you can take an idea
about how MiQ
links Hawkular metrics inside something that can be managed internally of
MiQ.
https://github.com/ManageIQ/manageiq/blob/master/product/live_metrics/mid...
https://github.com/ManageIQ/manageiq/blob/master/product/live_metrics/mid...
I guess that if the id of the type is still valid, then perhaps nothing
affects but just to be sure I chime in here just to validate it.
Right. This is OK. The comments in that MiQ code says "It maps the native id used in
the provider" - so this is OK because the native IDs (the metric type IDs) remain the
same as they are today, nothing changes (what you have there in the miq code -
"setName~typeName" is what the IDs will still be.)
The only thing changing is the type NAME (which will no longer be the same as the ID - I
strip the "setName~" from it). The reason for this change is because if a person
wants to define their own custom metric ID, I needed a way to give them the ability to
inject the metric name in their custom metric ID. They will most likely want to use the
metric name as part of the metric ID template - so I now support a token of
"%MetricTypeName" which users can use - it will be replaced with the name of the
metric type (not to include the "setName~" string).
11:08 p.m.
On středa 6. července 2016 21:13:37 CEST John Mazzitelli wrote:
<tl;dr> ======
Agent is introducing two changes:
1. Metric Type definitions created by the agent will have the same ID as
before, but its Name is changing (probably does not affect anyone).
2. Clients (like UI, HawkFX, etc) should no longer assume the agent's
h-inventory metric definition IDs match h-metric metric IDs - instead, they
must look at the "metric-id" property on the h-inventory metric definition
to know how to look up the actual metric data in h-metrics. Will affect all
clients, but metric IDs will default to what they are today - so nothing
changes and thus nothing will break today if you run with out of box
configuration.
</tl;dr> ======
There is a use-case where the agent needs to support custom metric IDs (that
is, rather than accepting the out-of-box metric IDs created by the agent,
allow the user to define what the metric IDs should look like). See
https://issues.jboss.org/browse/HWKAGENT-78
As a refresher, remember that when you create resources in inventory, those
resources can be associated with one or more "metric" definitions. Those
resource metrics are themselves associated with a "metric type" definition.
Today, when the agent stores metric data into Hawkular Metrics, it stores
the data under the ID of the "metric" that is associated with the resource
(so the h-inventory metric ID is the same as the h-metric metric ID by
definition - at least for the data the hawkular wildfly agent inserts).
I am proposing two changes in the PR:
https://github.com/hawkular/hawkular-agent/pull/226
First, today, the "metric type" definition that the agent creates has an ID
and a Name that are identical. I am changing this so the ID stays the same
(which is the metric set name, followed by "~", followed by the name of the
metric -- e.g. if there was a <metric-set-dmr name="this"> that contains
a
<metric-dmr name="that">, the metric type ID would be
"this~that"), but the
Name is only the name without the set name (e.g. the name would be "that"
in the previous example).
The above is a minor change, and I doubt anyone is affected by it. But I
point it out just in case.
Second, it should no longer be assumed that the inventory's resource metric
ID is identical to the h-metric's metric ID.
This second change will potentially affect everyone (I know it affects
Heiko's HawkFX :)
Now, that said, nothing really changes now, because the defaults will remain
as they are (that is, the h-inventory's metric ID will still be exactly the
same as the h-metrics ID - the agent keeps them identical). The change
happens when the user actually configures the agent with a custom metric ID
template (e.g. <remote-jmx metric-id-template="${pod.id}-%MetricTypeName"
...>). This means h-metric IDs will be DIFFERENT than h-inventory metric
IDs.
How then does a client know what h-metric IDs to look for if they only have
h-inventory metric definitions? Well, recall that inventory allows for
properties to be associated with any entity. I use this feature here.
Rather than rely on an implicit rule ("h-inventory metric ID is the same as
h-metric metric ID") I explicitly define this linkage in a property called
"metric-id" on the h-inventory metric definition. Out of box, that
property's value will be identical to the h-inventory metric ID (and hence
why nothing really changes - since the explicit rule in this case provides
the same behavior as if following the old implicit rule). In fact, I'm
considering if I should set that property at all if its the same as the
h-inventory ID - I think it might be better to only set a "metric-id"
property if it is different. But this would require clients to know about
the implicit rule if there is no metric-id property set ("is there a
metric-id property set? No? Then use the h-inven! tory metric ID for the
h-metric metric ID").
For example, see here (this is a live example I copied from the "raw"
inventory JSON that HawkFX gave me for a metric) - this is the h-inventory
entity definition for the metric "Heap Used" on my WildFly Server resource
- notice the "properties" map has a "metric-id" value that is
DIFFERENT
than the "id" - that "metric-id" is something I customized in my
agent
config in standalone.xml (well, I used the swarm agent, so I put it in the
swarm config, but its basically the same thing):
{
"path":
"/t;hawkular/f;mazz/m;MI~R~%5Bmazz%2FWildFly~~%5D~MT~WildFly%20Memory%20Met
rics~Heap%20Used", "properties": {
"__identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"metric-id": "feed=mazz__msn=WildFly__typeName=Heap
Used__resName=WildFly Server [WildFly]__resId=WildFly~~__typeId=WildFly
Memory Metrics~Heap Used" },
"name": "Heap Used",
"identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"type": {
"path":
"/t;hawkular/f;mazz/mt;WildFly%20Memory%20Metrics~Heap%20Used",
"name": "Heap Used",
"identityHash": "3be5b5fdabed925ac46fdc6d8295e34bbd3147a",
"unit": "BYTES",
"type": "GAUGE",
"collectionInterval": 30,
"id": "WildFly Memory Metrics~Heap Used"
},
"id": "MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used"
}
Notice this:
feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server
[WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used
is different from this:
MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used
And that's the issue. Clients have to know to look for the "metric-id"
property and use it when looking up metric data in h-metrics (so if you
want to graph the data, you have to ask h-metrics for the data associated
with the value found in the "metric-id" property).
What if you used the inventory's canonical path as an the metric ID for h-
metrics (e.g. used "path" from the json above)? After all, if inventory should
be the storage of all things Hawkular knows about, the canonical path is the
one and only thing that can identify any one thing in inventory. So it lends
itself nicely to be used as an identifier in other components.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
--
Lukas Krejci
11:31 p.m.
What if you used the inventory's canonical path as an the metric
ID
Well, that doesn't solve the problem. That would just be replacing one auto-generated
ID for another one. The requirement is to allow a user to be able to customize the metric
ID to something the user wants. See the JIRA for details:
https://issues.jboss.org/browse/HWKAGENT-78
11:53 p.m.
On čtvrtek 7. července 2016 17:31:14 CEST John Mazzitelli wrote:
> What if you used the inventory's canonical path as an the
metric ID
Well, that doesn't solve the problem. That would just be replacing one
auto-generated ID for another one. The requirement is to allow a user to be
able to customize the metric ID to something the user wants. See the JIRA
for details: https://issues.jboss.org/browse/HWKAGENT-78
Hmm, good point.
I wonder then if this should not evolve into something more engrained into
inventory. Something like "AKA" property on the entities:
"also-known-as": {
"h-metrics": "h-metrics-specific-id",
"h-alerts": "another-id",
"collectd": "yet-another-id",
"rhq": "some-id",
"legacy-monitoring-system-in-our-enterprise": "blah-id",
...
}
Well, maybe your approach of just shoving this info in general purpose
properties is enough. All the clients would still need to know where to look
for the other ids, so the above would possibly not bring too much
simplification for the clients. Consolidation maybe, but not simplification.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
--
Lukas Krejci
4:12 p.m.
On 7/7/2016 5:53 PM, Lukas Krejci wrote:
I wonder then if this should not evolve into something more engrained
into
inventory. Something like "AKA" property on the entities:
"also-known-as": {
"h-metrics": "h-metrics-specific-id",
"h-alerts": "another-id",
"collectd": "yet-another-id",
"rhq": "some-id",
"legacy-monitoring-system-in-our-enterprise": "blah-id",
...
}
This AKA property could be a good thing for mapping the Inventory metric
name to other IDs.
10:52 a.m.
Hi,
Why? The metricId is an id to a computer. It should not have anything meaningful to the
user. Use tags if you want some information to the stored id. Stop pushing
"structure" to the metricId, you can't use it for searching them.
- Micke
----- Original Message -----
From: "John Mazzitelli" <mazz(a)redhat.com>
To: hawkular-dev(a)lists.jboss.org
Sent: Friday, July 8, 2016 12:31:14 AM
Subject: Re: [Hawkular-dev] agent using custom metric IDs
What if you used the inventory's canonical path as an the metric
ID
Well, that doesn't solve the problem. That would just be replacing one auto-generated
ID for another one. The requirement is to allow a user to be able to customize the metric
ID to something the user wants. See the JIRA for details:
https://issues.jboss.org/browse/HWKAGENT-78
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
10:54 a.m.
+1
In general, id should be treated more from an abstract thing.
At the end, bringing some knowledge about how the id is built is a bad thing.
It is a technical debt that now we are suffering in MiQ and at some point we need to
address.
id should we similar as an UUID, IMO, nothing that you should use to extract business
info, for that we can include additional properties as Mike comments.
----- Mensaje original -----
De: "Michael Burman" <miburman(a)redhat.com>
Para: "John Mazzitelli" <mazz(a)redhat.com>, "Discussions around
Hawkular development" <hawkular-dev(a)lists.jboss.org>
Enviados: Viernes, 8 de Julio 2016 10:52:39
Asunto: Re: [Hawkular-dev] agent using custom metric IDs
Hi,
Why? The metricId is an id to a computer. It should not have anything
meaningful to the user. Use tags if you want some information to the stored
id. Stop pushing "structure" to the metricId, you can't use it for
searching
them.
- Micke
----- Original Message -----
From: "John Mazzitelli" <mazz(a)redhat.com>
To: hawkular-dev(a)lists.jboss.org
Sent: Friday, July 8, 2016 12:31:14 AM
Subject: Re: [Hawkular-dev] agent using custom metric IDs
> What if you used the inventory's canonical path as an the metric ID
Well, that doesn't solve the problem. That would just be replacing one
auto-generated ID for another one. The requirement is to allow a user to be
able to customize the metric ID to something the user wants. See the JIRA
for details: https://issues.jboss.org/browse/HWKAGENT-78
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2839
days inactive
2858
days old
8 comments
5 participants
participants (5)
-
Jay Shaughnessy -
John Mazzitelli -
Lucas Ponce -
Lukas Krejci -
Michael Burman