[Hawkular-dev] Inventory storage on metrics

[Joel Takvorian]

For information, the string sizes I have while running the agent's
integration tests is about 32KB uncompressed per metric (2.5KB gzipped), up
to 200KB (gzipped: 6.7KB). However I don't know if those numbers are of any
interest or if they will vary too much from real-world usage.

Joel

On Mon, Mar 6, 2017 at 8:18 PM, Joel Takvorian <jtakvori at redhat.com> wrote:

> I haven't checked that part of the puzzle yet, will do it tomorrow.
> However I hope it is possible to keep the ResourcePath as they are
> currently used. Since we can retrieve the root resource from a canonical
> path of a child resource, it's should be ok to find the desired associated
> string metric.
>
> On Mon, Mar 6, 2017 at 7:28 PM, John Mazzitelli <mazz at redhat.com> wrote:
>
>> This also affects the command-gateway stuff since virtually all of the
>> messages require an Inventory ResourcePath (that is how the MiQ UI targets
>> where it wants the message to go, and how the server knows what agent to
>> pass the message to, and how the agent knows what specific resource to
>> operate on).
>>
>> These ResourcePaths are Hawkular-Inventory resource paths - if resource
>> path formats change, we need to change the way the command-gateway stuff
>> works.
>>
>> I can't speak for the MiQ UI side, but from server/agent sides, it isn't
>> trivial but also not that hard to do these changes. It is just work we need
>> to remember we have to do - work in the MiQ UI, server, and agent.
>>
>> ----- Original Message -----
>> > Quick reminder for those who may not have followed the discussions:
>> this is
>> > about removing the inventory project as it currently stands, and
>> replace it
>> > with Metrics-based (so C*-based) storage, using String metrics and json
>> > blobs.
>> >
>> > So, I've been analyzing the usages of inventory in the wildfly agent
>> and in
>> > manageiq / ruby client.
>> >
>> > As posted on another topic, I started with the following pattern for
>> putting
>> > inventories in string metrics:
>> >
>> > - metrics are named:
>> > inventory.[feedid].[r|rt|mt].[id]
>> >
>> > which means that we would store JSON blobs by resource type, by metric
>> type
>> > and by root resource (all per feed id). Which also means we don't allow
>> > anything outside of a feed.
>> >
>> > This naming pattern seems to be quite easily transposable to the wildfly
>> > agent feeding.
>> >
>> > Things become more complicated when we look at the ruby client /
>> manageIQ. To
>> > help in querying, we can add those tags on every metric:
>> >
>> >
>> >
>> > module: inventory
>> >
>> > feed: [feedid]
>> >
>> > type: [r|rt|mt]
>> >
>> > With this, we can easily query, for instance, all resource types of a
>> given
>> > feed, or all feeds, or child resources at a given path, etc.
>> >
>> > Now, after analysis of the ruby code (and I hope I didn't miss important
>> > stuff), I have two remarks:
>> >
>> > - There is a lot a methods in the ruby client that are not used in
>> manage IQ
>> > (listed below [1]). Of course those methods may have an interest for a
>> > general purpose client, but since we're removing a lot of
>> functionalities in
>> > inventory, I would suggest not to focus on them, and just remove them
>> for
>> > the time being. This is possible, of course, if no other product than
>> > manageIQ consumes then... is it the case?
>> >
>> > - I've found just 1 method in the ruby client that would be badly
>> impacted by
>> > the usage of string metrics as currently defined. And unfortunately it's
>> > also the one that is most called (in number of occurrence in code) by
>> manage
>> > IQ, it's "list_resources_for_type"; With the implementation I described
>> > above, to list all resources of a given type we would have to get all
>> > resource blobs of a feed and search in all of them to check all root &
>> child
>> > resource types. That would probably be terrible for performances.
>> >
>> > A suggestion to this problem, is to tag each root resource with one tag
>> per
>> > resource type, something like: "
>> > rt.MyResourceType=resource1,resource2,resource3 ".
>> > Then, we would still have to load the whole blobs and search resources
>> in
>> > them, but we would exclude from the search all root resources that
>> doesn't
>> > have those types, and we wouldn't have to search extensively in the
>> blobs as
>> > we would know the paths.
>> >
>> > Question to metrics devs about that: can we use any character for
>> tagging, or
>> > is there some limitation?
>> >
>> > Or if you can suggest a better way, I'm all ears :)
>> >
>> > [1] list of methods that seem to be unused from the ruby client:
>> > - list_resources_for_feed
>> > - list_relationships
>> > - list_relationships_for_feed
>> > - get_entity
>> > - list_metrics_for_metric_type
>> > - list_metrics_for_resource_type
>> > - create_feed
>> > - delete_feed
>> > - create_resource_type
>> > - create_resource
>> > - create_resource_under_resource
>> > - create_metric_type
>> > - list_operation_definitions
>> > - list_operation_definitions_for_resource
>> > - create_metric_for_resource
>> > - no_more_events
>> >
>> > Joel
>> >
>> > _______________________________________________
>> > hawkular-dev mailing list
>> > hawkular-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/hawkular-dev
>> >
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170313/80d90c25/attachment.html