[Hawkular-dev] agent using custom metric IDs

[John Mazzitelli]

<tl;dr> ======

Agent is introducing two changes:

1. Metric Type definitions created by the agent will have the same ID as before, but its Name is changing (probably does not affect anyone).

2. Clients (like UI, HawkFX, etc) should no longer assume the agent's h-inventory metric definition IDs match h-metric metric IDs - instead, they must look at the "metric-id" property on the h-inventory metric definition to know how to look up the actual metric data in h-metrics. Will affect all clients, but metric IDs will default to what they are today - so nothing changes and thus nothing will break today if you run with out of box configuration.

</tl;dr> ======

There is a use-case where the agent needs to support custom metric IDs (that is, rather than accepting the out-of-box metric IDs created by the agent, allow the user to define what the metric IDs should look like). See https://issues.jboss.org/browse/HWKAGENT-78

As a refresher, remember that when you create resources in inventory, those resources can be associated with one or more "metric" definitions. Those resource metrics are themselves associated with a "metric type" definition. Today, when the agent stores metric data into Hawkular Metrics, it stores the data under the ID of the "metric" that is associated with the resource (so the h-inventory metric ID is the same as the h-metric metric ID by definition - at least for the data the hawkular wildfly agent inserts).

I am proposing two changes in the PR: https://github.com/hawkular/hawkular-agent/pull/226

First, today, the "metric type" definition that the agent creates has an ID and a Name that are identical. I am changing this so the ID stays the same (which is the metric set name, followed by "~", followed by the name of the metric -- e.g. if there was a <metric-set-dmr name="this"> that contains a <metric-dmr name="that">, the metric type ID would be "this~that"), but the Name is only the name without the set name (e.g. the name would be "that" in the previous example).

The above is a minor change, and I doubt anyone is affected by it. But I point it out just in case.

Second, it should no longer be assumed that the inventory's resource metric ID is identical to the h-metric's metric ID.

This second change will potentially affect everyone (I know it affects Heiko's HawkFX :)

Now, that said, nothing really changes now, because the defaults will remain as they are (that is, the h-inventory's metric ID will still be exactly the same as the h-metrics ID - the agent keeps them identical). The change happens when the user actually configures the agent with a custom metric ID template (e.g. <remote-jmx metric-id-template="${pod.id}-%MetricTypeName" ...>). This means h-metric IDs will be DIFFERENT than h-inventory metric IDs.

How then does a client know what h-metric IDs to look for if they only have h-inventory metric definitions? Well, recall that inventory allows for properties to be associated with any entity. I use this feature here. Rather than rely on an implicit rule ("h-inventory metric ID is the same as h-metric metric ID") I explicitly define this linkage in a property called "metric-id" on the h-inventory metric definition. Out of box, that property's value will be identical to the h-inventory metric ID (and hence why nothing really changes - since the explicit rule in this case provides the same behavior as if following the old implicit rule). In fact, I'm considering if I should set that property at all if its the same as the h-inventory ID - I think it might be better to only set a "metric-id" property if it is different. But this would require clients to know about the implicit rule if there is no metric-id property set ("is there a metric-id property set? No? Then use the h-inventory metric ID for the h-metric metric ID").

For example, see here (this is a live example I copied from the "raw" inventory JSON that HawkFX gave me for a metric) - this is the h-inventory entity definition for the metric "Heap Used" on my WildFly Server resource - notice the "properties" map has a "metric-id" value that is DIFFERENT than the "id" - that "metric-id" is something I customized in my agent config in standalone.xml (well, I used the swarm agent, so I put it in the swarm config, but its basically the same thing):

{
  "path": "/t;hawkular/f;mazz/m;MI~R~%5Bmazz%2FWildFly~~%5D~MT~WildFly%20Memory%20Metrics~Heap%20Used",
  "properties": {
    "__identityHash": "70e59a5d427632223da36c225ba6ef8572985",
    "metric-id": "feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server [WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used"
  },
  "name": "Heap Used",
  "identityHash": "70e59a5d427632223da36c225ba6ef8572985",
  "type": {
    "path": "/t;hawkular/f;mazz/mt;WildFly%20Memory%20Metrics~Heap%20Used",
    "name": "Heap Used",
    "identityHash": "3be5b5fdabed925ac46fdc6d8295e34bbd3147a",
    "unit": "BYTES",
    "type": "GAUGE",
    "collectionInterval": 30,
    "id": "WildFly Memory Metrics~Heap Used"
  },
  "id": "MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used"
}

Notice this:

   feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server [WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used

is different from this:

   MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used

And that's the issue. Clients have to know to look for the "metric-id" property and use it when looking up metric data in h-metrics (so if you want to graph the data, you have to ask h-metrics for the data associated with the value found in the "metric-id" property).