3 a.m.
Hey,
in RHQ we had several ways to record properties of things
* traits
* resource configuration
* plugin configuration
* ...
I think we need something similar in Hawkular.
Having said that, we probably should avoid the above situation
and find out what is really needed.
I can imagine that for stuff that are in reality changing attributes of
a resource (e.g. version, #of cpus, ip address, ....) - basically stuff
that is not changing like a metric, some key-value pairs on the
resource in inventory may be the best place.
8:03 a.m.
In my mind almost anything "is changing like a metric", because everything
changes over time.
Also IMHO "key-value pairs" are just JSON. I think that the structured data
support for configs and operation params served us well in RHQ.
So IMHO we need a time-series store for structured data. I am not sure inventory
is that - it is good for storing relationships among entities but I am not
sure if it is well suited for storing (versioned) structured data.
The reason for that is that such support would require somehow translating the
JSON-like format of the structured data into a graph of nodes in the underlying
graph database to enable any visibility of the constituent parts of the data
structure.
Alternatively, with Titan, it might be possible to achieve the same result
if we employed ElasticSearch as additional backend for it and configured it
to analyze the JSON. I have not yet looked into the possibilities, pros nor
cons of that approach.
Or just use ElasticSearch as a "properties store" straight away - both
inventory and IMHO metrics could benefit from having ES as part of the Hawkular
cluster for their own purposes but IMHO a dedicated properties store should
be geared towards storing, analyzing and querying structured data - which is
the exact sales pitch of ES ;)
Now I'm not saying that I know anything more about ES than I just said - maybe
it's completely different and I'm blabbering nonsense here, but I guess the
point I wanted to make is that inventory is not designed to store, query
or analyze structured data in and of itself.
Lukas
----- Original Message -----
From: "Heiko W.Rupp" <hrupp(a)redhat.com>
To: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Sent: Monday, 4 May, 2015 9:00:00 AM
Subject: [Hawkular-dev] Properties store?
Hey,
in RHQ we had several ways to record properties of things
* traits
* resource configuration
* plugin configuration
* ...
I think we need something similar in Hawkular.
Having said that, we probably should avoid the above situation
and find out what is really needed.
I can imagine that for stuff that are in reality changing attributes of
a resource (e.g. version, #of cpus, ip address, ....) - basically stuff
that is not changing like a metric, some key-value pairs on the
resource in inventory may be the best place.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
7:53 a.m.
On 4 May 2015, at 14:03, Lukas Krejci wrote:
In my mind almost anything "is changing like a metric",
because
everything changes over time.
Well. Then lets stop your project and make metrics store inventory
data :;->
Also IMHO "key-value pairs" are just JSON. I think that
the
structured data
support for configs and operation params served us well in RHQ.
I don't say that the structured data in RHQ is by itself bad (the
implementation
has IMO room for improvement). I also don't say that the traits in RHQ
are a bad thing.
But: with traits and configuration we have some very similar concepts
that have in RHQ some very different implementation.
So IMHO we need a time-series store for structured data. I am not
sure
inventory
is that - it is good for storing relationships among entities but I
am not
sure if it is well suited for storing (versioned) structured data.
I do not think that each single property needs to be a separate object.
I can imagine that we keep one (JSON) object per version (with some
additional
metadata) and the respective "client" of the store know how to deal with
it.
My understanding was that Inventory can have such a list of things
hanging on a resource.
The question remains though if inventory is the right place or metrics.
Also for Traits, I think we should not (like for Availability) store the
whole
string every 20sec, but only store an update when a change occurred (1),
which may or may be applicable for configuration too. In fact RHQ had
logic inside the agents to only send updates when it detected changes.
Also Traits are most likely r/o, while configuration data may be
modified from
the UI.
1) And perhaps the last time the old version was still detected.
9:01 a.m.
On Tuesday, May 19, 2015 13:53:23 Heiko W.Rupp wrote:
On 4 May 2015, at 14:03, Lukas Krejci wrote:
> In my mind almost anything "is changing like a metric", because
> everything changes over time.
Well. Then lets stop your project and make metrics store inventory
data :;->
> Also IMHO "key-value pairs" are just JSON. I think that the
> structured data
> support for configs and operation params served us well in RHQ.
I don't say that the structured data in RHQ is by itself bad (the
implementation
has IMO room for improvement). I also don't say that the traits in RHQ
are a bad thing.
But: with traits and configuration we have some very similar concepts
that have in RHQ some very different implementation.
> So IMHO we need a time-series store for structured data. I am not sure
> inventory
> is that - it is good for storing relationships among entities but I
> am not
> sure if it is well suited for storing (versioned) structured data.
I do not think that each single property needs to be a separate object.
I can imagine that we keep one (JSON) object per version (with some
additional
metadata) and the respective "client" of the store know how to deal with
it.
My understanding was that Inventory can have such a list of things
hanging on a resource.
Yes, it can, so in another words inventory can very well be used for storing
traits or config or whatever.
I think the main difference between inventory or metrics being used for
storing these is that of the focus of each component.
In my mind, inventory is primarily focused on displaying relationships and
properties of different entities *at some point in time*.
Metrics on the other hand is mainly focused on displaying the evolution of
values of a metric *over time*.
While you can certainly use inventory to build your view of changes over time,
it will probably not be optimized for that use-case. Likewise, if you would
like to use metrics to show you values of a number of metrics at some precise
point in time, it might not be optimized for that (and yes, this is a wild
guess of someone who doesn't know metrics at all ;) ).
But in the end, if we decide inventory is the best candidate for doing this, I
hope it can be bent to do what we need in some reasonable way.
The question remains though if inventory is the right place or
metrics.
Also for Traits, I think we should not (like for Availability) store the
whole
string every 20sec, but only store an update when a change occurred (1),
which may or may be applicable for configuration too. In fact RHQ had
logic inside the agents to only send updates when it detected changes.
Also Traits are most likely r/o, while configuration data may be
modified from
the UI.
1) And perhaps the last time the old version was still detected.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
9:16 a.m.
On May 19, 2015, at 7:53 AM, Heiko W.Rupp <hrupp(a)redhat.com>
wrote:
On 4 May 2015, at 14:03, Lukas Krejci wrote:
> In my mind almost anything "is changing like a metric", because
> everything changes over time.
Well. Then lets stop your project and make metrics store inventory
data :;->
I agree with Lukas. At lease in RHQ, we treated most things as time series because we are
interested in changes over time. Metrics is one type of time series.
> Also IMHO "key-value pairs" are just JSON. I think that the
> structured data
> support for configs and operation params served us well in RHQ.
I don't say that the structured data in RHQ is by itself bad (the
implementation
has IMO room for improvement). I also don't say that the traits in RHQ
are a bad thing.
But: with traits and configuration we have some very similar concepts
that have in RHQ some very different implementation.
> So IMHO we need a time-series store for structured data. I am not sure
> inventory
> is that - it is good for storing relationships among entities but I
> am not
> sure if it is well suited for storing (versioned) structured data.
I do not think that each single property needs to be a separate object.
I can imagine that we keep one (JSON) object per version (with some
additional
metadata) and the respective "client" of the store know how to deal with
it.
My understanding was that Inventory can have such a list of things
hanging on a resource.
The question remains though if inventory is the right place or metrics.
Also for Traits, I think we should not (like for Availability) store the
whole
string every 20sec, but only store an update when a change occurred (1),
which may or may be applicable for configuration too. In fact RHQ had
logic inside the agents to only send updates when it detected changes.
Also Traits are most likely r/o, while configuration data may be
modified from
the UI.
1) And perhaps the last time the old version was still detected.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
8:13 a.m.
On 19 May 2015, at 15:16, John Sanda wrote:
I agree with Lukas. At lease in RHQ, we treated most things as time
series because we are interested in changes over time. Metrics is one
type of time series.
Of course we are interested in changes over time.
That does not imply storing everything as metric, as (to use that
argument too) also did not do in RHQ.
Traits and configuration data got a very separate treatment from
numeric metrics and also need it.
How do you aggregate traits or config objects?
How do you display traits or config objects in charts for gauge data?
11:23 p.m.
On May 20, 2015, at 8:13 AM, Heiko W.Rupp <hrupp(a)redhat.com>
wrote:
On 19 May 2015, at 15:16, John Sanda wrote:
> I agree with Lukas. At lease in RHQ, we treated most things as time
> series because we are interested in changes over time. Metrics is one
> type of time series.
Of course we are interested in changes over time.
That does not imply storing everything as metric, as (to use that
argument too) also did not do in RHQ.
Traits and configuration data got a very separate treatment from
numeric metrics and also need it.
How do you aggregate traits or config objects?
How do you display traits or config objects in charts for gauge data?
I think you misunderstand what I am suggesting. Think more along the lines of a version
control system. Every piece of data that gets persisted has a timestamp associated with
it. In general, updating data means persisting a new copy of the data with a new
timestamp. Cassandra excels at handling time series data, and again, I am not just talking
about metrics. It excels because of the read/write patterns that are common with time
series. In RHQ we treated most things as time series as evidence from all of the History
entities. Whether it is a trait or some resource configuration, we do not want to store
just the latest value, but the changes over time.
8:42 a.m.
Think more along the
lines of a version control system. Every piece of data that gets
persisted has a timestamp associated with it. In general, updating
This is exactly what I am thinking of.
Cassandra excels at handling time series data, and again, I am not
just talking about metrics. It excels because of the read/write
patterns that are common with time series. In RHQ we treated most
things as time series as evidence from all of the History entities.
Whether it is a trait or some resource configuration, we do not want
to store just the latest value, but the changes over time.
I fully agree here.
And I am not questioning the use of Cassandra for this at all.
In fact Inventory uses C* behind the scenes as well.
My point is rather that traits and configurations are very similar to
each other,
while trait and numeric metric are less similar.
So the question is still, how and where to deal with that.
If we assume that configuration and traits are just ("arbitrary" long)
strings,
we need
- de-duplication - we only want to save changes, not every value when it
is
the same as the previous version
- making sure when expiring after some time, that there is always
something
left to look at (the value that was first recorded may have never
changed since)
- probably make sure we record (from time to time) the last time the
value was seen
"Consumers" on top can then decide that the contents is actually a json-
structure, and translate it to json and a Map to deal with it.
While my gut feeling is that configuration data is "closer" to a
resource, than
traits, I think at the end both are just one URI away. We could have
another
component to store those, but having a "string store" inside the metrics
project
that satisfies the above requirements (and probably more), is probably
enough.
10:46 a.m.
I know I yell "Artificer!" a lot ;), but this is another central use case.
Store 1 logical, metadata-only artifact per resource, then add in the key/value
properties.
----- Original Message -----
From: "Heiko W.Rupp" <hrupp(a)redhat.com>
To: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Sent: Monday, May 4, 2015 3:00:00 AM
Subject: [Hawkular-dev] Properties store?
Hey,
in RHQ we had several ways to record properties of things
* traits
* resource configuration
* plugin configuration
* ...
I think we need something similar in Hawkular.
Having said that, we probably should avoid the above situation
and find out what is really needed.
I can imagine that for stuff that are in reality changing attributes of
a resource (e.g. version, #of cpus, ip address, ....) - basically stuff
that is not changing like a metric, some key-value pairs on the
resource in inventory may be the best place.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
3302
days inactive
3319
days old
8 comments
4 participants
participants (4)
-
Brett Meyer -
Heiko W.Rupp -
John Sanda -
Lukas Krejci