midnight
<tl;dr>
Need ideas on how we are to implement the following two things in the agent:
1. At startup, agent needs to ask H-Metrics "what top-level servers have I told you
about in an earlier life?"
2. When a new server is discovered, the agent should send an event to the server about the
new server EXCEPT if the server isn't really new at all (see 1. above)
</tl;dr>
===
This post is to open up a discussion on how we want to implement a new features in the
agent.
Joel is developing a new "inventory in metrics" feature:
https://github.com/hawkular/hawkular-agent/pull/303
This means the agent will be storing inventory directly into Hawkular-Metrics. Because of
this, we need to figure out how to get events sent based on things happening in
H-Metric's inventory so MiQ can do things with it (like put things in the timeline
such as "new server discovered" or "new WAR was deployed").
Jay looked at the code and the only thing that would be "missing" after this
move of inventory into metrics is an event triggered when a new server is added to
inventory. (When a new deployment is added, or a deployment is removed, the server is
looking at command responses and generating events from that - so we don't lose
anything by moving inventory into metrics).
By "new server", what we mean is a new resource that has no parent resources
(i.e. a "root resource"). This includes standalone WildFly Servers and domain
Host Controllers.
Right now, the agent starts with a "clean slate" when it starts up for the first
time, or restarts. That means the agent's in-memory inventory graph is completely
empty at startup - when discovery is run, the agent's internal inventory graph is
filled in. After that, the agent just keeps the inventory graph up to date as it discovers
new things coming and old things going away.
We need the agent to know if it already stored its top level servers into H-Metrics
inventory and if it did, not to generate any "new server event". But if the
agent is brand new, and it never sent any top-level resources to H-Metrics inventory yet,
it should now send a "new server" event to the server (the agent never sent
events like this before).
So there are two new things (assuming we keep the stuff Joel is doing - that is, we store
inventory into H-Metrics):
1. At startup, agent needs to ask H-Metrics "what top-level servers have I told you
about in an earlier life?"
2. When a new server is discovered, the agent should send an event (whatever this means -
probably a REST API call somewhere) about the new server EXCEPT if the server isn't
really new at all (see 1. above)
We need to figure out how to implement 1. and 2. So we are soliciting thoughts on those
two subjects.
1:40 a.m.
For point 1., we can probably use some functions I wrote in the integration
tests, see there:
https://github.com/jotak/hawkular-agent/blob/inventory-strings/hawkular-a...
I assume you can build a canonical path? (The same "canonical path" than in
the existing inventory) If so, the method "getBlueprintFromCP" gives it to
you as an Optional blueprint.
On Mon, Mar 27, 2017 at 5:00 PM, John Mazzitelli <mazz(a)redhat.com> wrote:
<tl;dr>
Need ideas on how we are to implement the following two things in the
agent:
1. At startup, agent needs to ask H-Metrics "what top-level servers have I
told you about in an earlier life?"
2. When a new server is discovered, the agent should send an event to the
server about the new server EXCEPT if the server isn't really new at all
(see 1. above)
</tl;dr>
===
This post is to open up a discussion on how we want to implement a new
features in the agent.
Joel is developing a new "inventory in metrics" feature:
https://github.com/hawkular/hawkular-agent/pull/303
This means the agent will be storing inventory directly into
Hawkular-Metrics. Because of this, we need to figure out how to get events
sent based on things happening in H-Metric's inventory so MiQ can do things
with it (like put things in the timeline such as "new server discovered" or
"new WAR was deployed").
Jay looked at the code and the only thing that would be "missing" after
this move of inventory into metrics is an event triggered when a new server
is added to inventory. (When a new deployment is added, or a deployment is
removed, the server is looking at command responses and generating events
from that - so we don't lose anything by moving inventory into metrics).
By "new server", what we mean is a new resource that has no parent
resources (i.e. a "root resource"). This includes standalone WildFly
Servers and domain Host Controllers.
Right now, the agent starts with a "clean slate" when it starts up for the
first time, or restarts. That means the agent's in-memory inventory graph
is completely empty at startup - when discovery is run, the agent's
internal inventory graph is filled in. After that, the agent just keeps the
inventory graph up to date as it discovers new things coming and old things
going away.
We need the agent to know if it already stored its top level servers into
H-Metrics inventory and if it did, not to generate any "new server event".
But if the agent is brand new, and it never sent any top-level resources to
H-Metrics inventory yet, it should now send a "new server" event to the
server (the agent never sent events like this before).
So there are two new things (assuming we keep the stuff Joel is doing -
that is, we store inventory into H-Metrics):
1. At startup, agent needs to ask H-Metrics "what top-level servers have I
told you about in an earlier life?"
2. When a new server is discovered, the agent should send an event
(whatever this means - probably a REST API call somewhere) about the new
server EXCEPT if the server isn't really new at all (see 1. above)
We need to figure out how to implement 1. and 2. So we are soliciting
thoughts on those two subjects.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
6:02 a.m.
i was thinking, perhaps it's not even necessary to involve inventory to
know if the server was reported or not. We define metrics, like avail,
for these root types, I think. At startup if the metric existed perhaps
you could assume it was already reported, otherwise you could send a
"new server" event. Would that approach fly or be easier?
On 3/27/2017 12:40 PM, Joel Takvorian wrote:
For point 1., we can probably use some functions I wrote in the
integration tests, see there:
https://github.com/jotak/hawkular-agent/blob/inventory-strings/hawkular-a...
I assume you can build a canonical path? (The same "canonical path"
than in the existing inventory) If so, the method "getBlueprintFromCP"
gives it to you as an Optional blueprint.
On Mon, Mar 27, 2017 at 5:00 PM, John Mazzitelli <mazz(a)redhat.com
<mailto:mazz@redhat.com>> wrote:
<tl;dr>
Need ideas on how we are to implement the following two things in
the agent:
1. At startup, agent needs to ask H-Metrics "what top-level
servers have I told you about in an earlier life?"
2. When a new server is discovered, the agent should send an event
to the server about the new server EXCEPT if the server isn't
really new at all (see 1. above)
</tl;dr>
===
This post is to open up a discussion on how we want to implement a
new features in the agent.
Joel is developing a new "inventory in metrics" feature:
https://github.com/hawkular/hawkular-agent/pull/303
<https://github.com/hawkular/hawkular-agent/pull/303>
This means the agent will be storing inventory directly into
Hawkular-Metrics. Because of this, we need to figure out how to
get events sent based on things happening in H-Metric's inventory
so MiQ can do things with it (like put things in the timeline such
as "new server discovered" or "new WAR was deployed").
Jay looked at the code and the only thing that would be "missing"
after this move of inventory into metrics is an event triggered
when a new server is added to inventory. (When a new deployment is
added, or a deployment is removed, the server is looking at
command responses and generating events from that - so we don't
lose anything by moving inventory into metrics).
By "new server", what we mean is a new resource that has no parent
resources (i.e. a "root resource"). This includes standalone
WildFly Servers and domain Host Controllers.
Right now, the agent starts with a "clean slate" when it starts up
for the first time, or restarts. That means the agent's in-memory
inventory graph is completely empty at startup - when discovery is
run, the agent's internal inventory graph is filled in. After
that, the agent just keeps the inventory graph up to date as it
discovers new things coming and old things going away.
We need the agent to know if it already stored its top level
servers into H-Metrics inventory and if it did, not to generate
any "new server event". But if the agent is brand new, and it
never sent any top-level resources to H-Metrics inventory yet, it
should now send a "new server" event to the server (the agent
never sent events like this before).
So there are two new things (assuming we keep the stuff Joel is
doing - that is, we store inventory into H-Metrics):
1. At startup, agent needs to ask H-Metrics "what top-level
servers have I told you about in an earlier life?"
2. When a new server is discovered, the agent should send an event
(whatever this means - probably a REST API call somewhere) about
the new server EXCEPT if the server isn't really new at all (see
1. above)
We need to figure out how to implement 1. and 2. So we are
soliciting thoughts on those two subjects.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
12:36 p.m.
My first thought was a string metric where data points are the servers that get
discovered. There are a couple things though that I do not like about this. First, all
data point queries in hawkular-metrics have a date range. Having to use a date range here
seems a bit awkward. Secondly all data points in hawkular-metrics expire. These does not
seem like data that we would want to expire.
On Mar 27, 2017, at 5:02 PM, Jay Shaughnessy
<jshaughn(a)redhat.com> wrote:
i was thinking, perhaps it's not even necessary to involve inventory to know if the
server was reported or not. We define metrics, like avail, for these root types, I think.
At startup if the metric existed perhaps you could assume it was already reported,
otherwise you could send a "new server" event. Would that approach fly or be
easier?
On 3/27/2017 12:40 PM, Joel Takvorian wrote:
> For point 1., we can probably use some functions I wrote in the integration tests,
see there:
https://github.com/jotak/hawkular-agent/blob/inventory-strings/hawkular-a...
<https://github.com/jotak/hawkular-agent/blob/inventory-strings/hawkular-a...
>
> I assume you can build a canonical path? (The same "canonical path" than in
the existing inventory) If so, the method "getBlueprintFromCP" gives it to you
as an Optional blueprint.
>
> On Mon, Mar 27, 2017 at 5:00 PM, John Mazzitelli <mazz(a)redhat.com
<mailto:mazz@redhat.com>> wrote:
> <tl;dr>
>
> Need ideas on how we are to implement the following two things in the agent:
>
> 1. At startup, agent needs to ask H-Metrics "what top-level servers have I told
you about in an earlier life?"
>
> 2. When a new server is discovered, the agent should send an event to the server
about the new server EXCEPT if the server isn't really new at all (see 1. above)
>
> </tl;dr>
>
>
> ===
>
>
> This post is to open up a discussion on how we want to implement a new features in
the agent.
>
> Joel is developing a new "inventory in metrics" feature:
https://github.com/hawkular/hawkular-agent/pull/303
<https://github.com/hawkular/hawkular-agent/pull/303>
>
> This means the agent will be storing inventory directly into Hawkular-Metrics.
Because of this, we need to figure out how to get events sent based on things happening in
H-Metric's inventory so MiQ can do things with it (like put things in the timeline
such as "new server discovered" or "new WAR was deployed").
>
> Jay looked at the code and the only thing that would be "missing" after
this move of inventory into metrics is an event triggered when a new server is added to
inventory. (When a new deployment is added, or a deployment is removed, the server is
looking at command responses and generating events from that - so we don't lose
anything by moving inventory into metrics).
>
> By "new server", what we mean is a new resource that has no parent
resources (i.e. a "root resource"). This includes standalone WildFly Servers and
domain Host Controllers.
>
> Right now, the agent starts with a "clean slate" when it starts up for the
first time, or restarts. That means the agent's in-memory inventory graph is
completely empty at startup - when discovery is run, the agent's internal inventory
graph is filled in. After that, the agent just keeps the inventory graph up to date as it
discovers new things coming and old things going away.
>
> We need the agent to know if it already stored its top level servers into H-Metrics
inventory and if it did, not to generate any "new server event". But if the
agent is brand new, and it never sent any top-level resources to H-Metrics inventory yet,
it should now send a "new server" event to the server (the agent never sent
events like this before).
>
> So there are two new things (assuming we keep the stuff Joel is doing - that is, we
store inventory into H-Metrics):
>
> 1. At startup, agent needs to ask H-Metrics "what top-level servers have I told
you about in an earlier life?"
>
> 2. When a new server is discovered, the agent should send an event (whatever this
means - probably a REST API call somewhere) about the new server EXCEPT if the server
isn't really new at all (see 1. above)
>
> We need to figure out how to implement 1. and 2. So we are soliciting thoughts on
those two subjects.
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
10:25 p.m.
If we could just use existing metrics to make the determination we could
avoid the TTL issue, I think. If we went with something like below I'd
just suggest a single metric definition with a tag for each server
name. Do metric defs suffer from TTL?
On 3/27/2017 11:36 PM, John Sanda wrote:
My first thought was a string metric where data points are the
servers
that get discovered. There are a couple things though that I do not
like about this. First, all data point queries in hawkular-metrics
have a date range. Having to use a date range here seems a bit
awkward. Secondly all data points in hawkular-metrics expire. These
does not seem like data that we would want to expire.
> On Mar 27, 2017, at 5:02 PM, Jay Shaughnessy <jshaughn(a)redhat.com
> <mailto:jshaughn@redhat.com>> wrote:
>
>
> i was thinking, perhaps it's not even necessary to involve inventory
> to know if the server was reported or not. We define metrics, like
> avail, for these root types, I think. At startup if the metric
> existed perhaps you could assume it was already reported, otherwise
> you could send a "new server" event. Would that approach fly or be
> easier?
>
> On 3/27/2017 12:40 PM, Joel Takvorian wrote:
>> For point 1., we can probably use some functions I wrote in the
>> integration tests, see there:
>>
https://github.com/jotak/hawkular-agent/blob/inventory-strings/hawkular-a...
>>
>>
>> I assume you can build a canonical path? (The same "canonical path"
>> than in the existing inventory) If so, the method
>> "getBlueprintFromCP" gives it to you as an Optional blueprint.
>>
>> On Mon, Mar 27, 2017 at 5:00 PM, John Mazzitelli <mazz(a)redhat.com
>> <mailto:mazz@redhat.com>> wrote:
>>
>> <tl;dr>
>>
>> Need ideas on how we are to implement the following two things
>> in the agent:
>>
>> 1. At startup, agent needs to ask H-Metrics "what top-level
>> servers have I told you about in an earlier life?"
>>
>> 2. When a new server is discovered, the agent should send an
>> event to the server about the new server EXCEPT if the server
>> isn't really new at all (see 1. above)
>>
>> </tl;dr>
>>
>>
>> ===
>>
>>
>> This post is to open up a discussion on how we want to implement
>> a new features in the agent.
>>
>> Joel is developing a new "inventory in metrics" feature:
>> https://github.com/hawkular/hawkular-agent/pull/303
>> <https://github.com/hawkular/hawkular-agent/pull/303>
>>
>> This means the agent will be storing inventory directly into
>> Hawkular-Metrics. Because of this, we need to figure out how to
>> get events sent based on things happening in H-Metric's
>> inventory so MiQ can do things with it (like put things in the
>> timeline such as "new server discovered" or "new WAR was
deployed").
>>
>> Jay looked at the code and the only thing that would be
>> "missing" after this move of inventory into metrics is an event
>> triggered when a new server is added to inventory. (When a new
>> deployment is added, or a deployment is removed, the server is
>> looking at command responses and generating events from that -
>> so we don't lose anything by moving inventory into metrics).
>>
>> By "new server", what we mean is a new resource that has no
>> parent resources (i.e. a "root resource"). This includes
>> standalone WildFly Servers and domain Host Controllers.
>>
>> Right now, the agent starts with a "clean slate" when it starts
>> up for the first time, or restarts. That means the agent's
>> in-memory inventory graph is completely empty at startup - when
>> discovery is run, the agent's internal inventory graph is filled
>> in. After that, the agent just keeps the inventory graph up to
>> date as it discovers new things coming and old things going away.
>>
>> We need the agent to know if it already stored its top level
>> servers into H-Metrics inventory and if it did, not to generate
>> any "new server event". But if the agent is brand new, and it
>> never sent any top-level resources to H-Metrics inventory yet,
>> it should now send a "new server" event to the server (the agent
>> never sent events like this before).
>>
>> So there are two new things (assuming we keep the stuff Joel is
>> doing - that is, we store inventory into H-Metrics):
>>
>> 1. At startup, agent needs to ask H-Metrics "what top-level
>> servers have I told you about in an earlier life?"
>>
>> 2. When a new server is discovered, the agent should send an
>> event (whatever this means - probably a REST API call somewhere)
>> about the new server EXCEPT if the server isn't really new at
>> all (see 1. above)
>>
>> We need to figure out how to implement 1. and 2. So we are
>> soliciting thoughts on those two subjects.
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>> <https://lists.jboss.org/mailman/listinfo/hawkular-dev>
>>
>>
>>
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
11:09 p.m.
The metric definitions do not expire, but they need to be explicitly created for what you
are suggesting to work. Even then we might run into problems. We are introducing a feature
to purge metric definitions that have not received any data for some time. Unless we
continue to periodically update the metrics used, we might run into problems.
On Mar 28, 2017, at 9:25 AM, Jay Shaughnessy
<jshaughn(a)redhat.com> wrote:
If we could just use existing metrics to make the determination we could avoid the TTL
issue, I think. If we went with something like below I'd just suggest a single
metric definition with a tag for each server name. Do metric defs suffer from TTL?
On 3/27/2017 11:36 PM, John Sanda wrote:
> My first thought was a string metric where data points are the servers that get
discovered. There are a couple things though that I do not like about this. First, all
data point queries in hawkular-metrics have a date range. Having to use a date range here
seems a bit awkward. Secondly all data points in hawkular-metrics expire. These does not
seem like data that we would want to expire.
>
>> On Mar 27, 2017, at 5:02 PM, Jay Shaughnessy <jshaughn(a)redhat.com
<mailto:jshaughn@redhat.com>> wrote:
>>
>>
>> i was thinking, perhaps it's not even necessary to involve inventory to know
if the server was reported or not. We define metrics, like avail, for these root types, I
think. At startup if the metric existed perhaps you could assume it was already reported,
otherwise you could send a "new server" event. Would that approach fly or be
easier?
>>
>> On 3/27/2017 12:40 PM, Joel Takvorian wrote:
>>> For point 1., we can probably use some functions I wrote in the integration
tests, see there:
https://github.com/jotak/hawkular-agent/blob/inventory-strings/hawkular-a...
<https://github.com/jotak/hawkular-agent/blob/inventory-strings/hawkular-a...
>>>
>>> I assume you can build a canonical path? (The same "canonical path"
than in the existing inventory) If so, the method "getBlueprintFromCP" gives it
to you as an Optional blueprint.
>>>
>>> On Mon, Mar 27, 2017 at 5:00 PM, John Mazzitelli <mazz(a)redhat.com
<mailto:mazz@redhat.com>> wrote:
>>> <tl;dr>
>>>
>>> Need ideas on how we are to implement the following two things in the agent:
>>>
>>> 1. At startup, agent needs to ask H-Metrics "what top-level servers have
I told you about in an earlier life?"
>>>
>>> 2. When a new server is discovered, the agent should send an event to the
server about the new server EXCEPT if the server isn't really new at all (see 1.
above)
>>>
>>> </tl;dr>
>>>
>>>
>>> ===
>>>
>>>
>>> This post is to open up a discussion on how we want to implement a new
features in the agent.
>>>
>>> Joel is developing a new "inventory in metrics" feature:
https://github.com/hawkular/hawkular-agent/pull/303
<https://github.com/hawkular/hawkular-agent/pull/303>
>>>
>>> This means the agent will be storing inventory directly into
Hawkular-Metrics. Because of this, we need to figure out how to get events sent based on
things happening in H-Metric's inventory so MiQ can do things with it (like put things
in the timeline such as "new server discovered" or "new WAR was
deployed").
>>>
>>> Jay looked at the code and the only thing that would be "missing"
after this move of inventory into metrics is an event triggered when a new server is added
to inventory. (When a new deployment is added, or a deployment is removed, the server is
looking at command responses and generating events from that - so we don't lose
anything by moving inventory into metrics).
>>>
>>> By "new server", what we mean is a new resource that has no parent
resources (i.e. a "root resource"). This includes standalone WildFly Servers and
domain Host Controllers.
>>>
>>> Right now, the agent starts with a "clean slate" when it starts up
for the first time, or restarts. That means the agent's in-memory inventory graph is
completely empty at startup - when discovery is run, the agent's internal inventory
graph is filled in. After that, the agent just keeps the inventory graph up to date as it
discovers new things coming and old things going away.
>>>
>>> We need the agent to know if it already stored its top level servers into
H-Metrics inventory and if it did, not to generate any "new server event". But
if the agent is brand new, and it never sent any top-level resources to H-Metrics
inventory yet, it should now send a "new server" event to the server (the agent
never sent events like this before).
>>>
>>> So there are two new things (assuming we keep the stuff Joel is doing - that
is, we store inventory into H-Metrics):
>>>
>>> 1. At startup, agent needs to ask H-Metrics "what top-level servers have
I told you about in an earlier life?"
>>>
>>> 2. When a new server is discovered, the agent should send an event (whatever
this means - probably a REST API call somewhere) about the new server EXCEPT if the server
isn't really new at all (see 1. above)
>>>
>>> We need to figure out how to implement 1. and 2. So we are soliciting
thoughts on those two subjects.
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
>>>
>>>
>>>
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
11:43 p.m.
And btw I have the exact same problem with the inventory storage on metrics
On Tue, Mar 28, 2017 at 4:09 PM, John Sanda <jsanda(a)redhat.com> wrote:
The metric definitions do not expire, but they need to be explicitly
created for what you are suggesting to work. Even then we might run into
problems. We are introducing a feature to purge metric definitions that
have not received any data for some time. Unless we continue to
periodically update the metrics used, we might run into problems.
On Mar 28, 2017, at 9:25 AM, Jay Shaughnessy <jshaughn(a)redhat.com> wrote:
If we could just use existing metrics to make the determination we could
avoid the TTL issue, I think. If we went with something like below I'd
just suggest a single metric definition with a tag for each server name.
Do metric defs suffer from TTL?
On 3/27/2017 11:36 PM, John Sanda wrote:
My first thought was a string metric where data points are the servers
that get discovered. There are a couple things though that I do not like
about this. First, all data point queries in hawkular-metrics have a date
range. Having to use a date range here seems a bit awkward. Secondly all
data points in hawkular-metrics expire. These does not seem like data that
we would want to expire.
On Mar 27, 2017, at 5:02 PM, Jay Shaughnessy <jshaughn(a)redhat.com> wrote:
i was thinking, perhaps it's not even necessary to involve inventory to
know if the server was reported or not. We define metrics, like avail, for
these root types, I think. At startup if the metric existed perhaps you
could assume it was already reported, otherwise you could send a "new
server" event. Would that approach fly or be easier?
On 3/27/2017 12:40 PM, Joel Takvorian wrote:
For point 1., we can probably use some functions I wrote in the
integration tests, see there: https://github.com/
jotak/hawkular-agent/blob/inventory-strings/hawkular-
agent-itest-util/src/main/java/org/hawkular/agent/itest/
util/ITestHelper.java
I assume you can build a canonical path? (The same "canonical path" than
in the existing inventory) If so, the method "getBlueprintFromCP" gives
it to you as an Optional blueprint.
On Mon, Mar 27, 2017 at 5:00 PM, John Mazzitelli <mazz(a)redhat.com> wrote:
> <tl;dr>
>
> Need ideas on how we are to implement the following two things in the
> agent:
>
> 1. At startup, agent needs to ask H-Metrics "what top-level servers have
> I told you about in an earlier life?"
>
> 2. When a new server is discovered, the agent should send an event to the
> server about the new server EXCEPT if the server isn't really new at all
> (see 1. above)
>
> </tl;dr>
>
>
> ===
>
>
> This post is to open up a discussion on how we want to implement a new
> features in the agent.
>
> Joel is developing a new "inventory in metrics" feature:
> https://github.com/hawkular/hawkular-agent/pull/303
>
> This means the agent will be storing inventory directly into
> Hawkular-Metrics. Because of this, we need to figure out how to get events
> sent based on things happening in H-Metric's inventory so MiQ can do things
> with it (like put things in the timeline such as "new server discovered"
or
> "new WAR was deployed").
>
> Jay looked at the code and the only thing that would be "missing" after
> this move of inventory into metrics is an event triggered when a new server
> is added to inventory. (When a new deployment is added, or a deployment is
> removed, the server is looking at command responses and generating events
> from that - so we don't lose anything by moving inventory into metrics).
>
> By "new server", what we mean is a new resource that has no parent
> resources (i.e. a "root resource"). This includes standalone WildFly
> Servers and domain Host Controllers.
>
> Right now, the agent starts with a "clean slate" when it starts up for
> the first time, or restarts. That means the agent's in-memory inventory
> graph is completely empty at startup - when discovery is run, the agent's
> internal inventory graph is filled in. After that, the agent just keeps the
> inventory graph up to date as it discovers new things coming and old things
> going away.
>
> We need the agent to know if it already stored its top level servers into
> H-Metrics inventory and if it did, not to generate any "new server event".
> But if the agent is brand new, and it never sent any top-level resources to
> H-Metrics inventory yet, it should now send a "new server" event to the
> server (the agent never sent events like this before).
>
> So there are two new things (assuming we keep the stuff Joel is doing -
> that is, we store inventory into H-Metrics):
>
> 1. At startup, agent needs to ask H-Metrics "what top-level servers have
> I told you about in an earlier life?"
>
> 2. When a new server is discovered, the agent should send an event
> (whatever this means - probably a REST API call somewhere) about the new
> server EXCEPT if the server isn't really new at all (see 1. above)
>
> We need to figure out how to implement 1. and 2. So we are soliciting
> thoughts on those two subjects.
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
_______________________________________________
hawkular-dev mailing
listhawkular-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing
listhawkular-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
5:35 p.m.
On Tue, Mar 28, 2017 at 4:09 PM, John Sanda <jsanda(a)redhat.com> wrote:
We are introducing a feature to purge metric definitions that have
not
received any data for some time.
If I understand this PR [1], metrics are eligible for deletion only when
their data points have all expired. If it's true then fixing the TTL
problem also fixes the metrics expiration problem.
[1] https://github.com/hawkular/hawkular-metrics/pull/785
9:29 p.m.
On Mar 29, 2017, at 4:35 AM, Joel Takvorian
<jtakvori(a)redhat.com> wrote:
On Tue, Mar 28, 2017 at 4:09 PM, John Sanda <jsanda(a)redhat.com
<mailto:jsanda@redhat.com>> wrote:
We are introducing a feature to purge metric definitions that have not received any data
for some time.
If I understand this PR [1], metrics are eligible for deletion only when their data
points have all expired. If it's true then fixing the TTL problem also fixes the
metrics expiration problem.
[1] https://github.com/hawkular/hawkular-metrics/pull/785
<https://github.com/hawkular/hawkular-metrics/pull/785>
Prior to this PR we
added support for metric deletion; however, deletion does not result in immediate deletion
of data points. We still let data points in in the data_compressed table expire. We do
this because one of the guidelines for using TimeWindowCompactionStrategy (TWCS) is to use
a global TTL and not delete data.
10:22 p.m.
On 29 Mar 2017, at 13:29, John Sanda wrote:
however, deletion does not result in immediate deletion of data
points. We still let data points in in the data_compressed table
expire. We do this because one of the guidelines for using
TimeWindowCompactionStrategy (TWCS) is to use a global TTL and not
delete data._______________________________________________
This needs to be well documented as this is not what you
expect when you delete metrics in an urgency to free up space.
11:01 p.m.
On Wed, Mar 29, 2017 at 3:35 AM, Joel Takvorian <jtakvori(a)redhat.com> wrote:
On Tue, Mar 28, 2017 at 4:09 PM, John Sanda <jsanda(a)redhat.com>
wrote:
> We are introducing a feature to purge metric definitions that have not
> received any data for some time.
>
>
One more important detail about the PR, we will not yet purge String metric
definitions. We will enable String purge in a future release.
If I understand this PR [1], metrics are eligible for deletion only
when
their data points have all expired. If it's true then fixing the TTL
problem also fixes the metrics expiration problem.
[1] https://github.com/hawkular/hawkular-metrics/pull/785
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
11:22 p.m.
On Mar 29, 2017, at 9:22 AM, Heiko W.Rupp <hrupp(a)redhat.com>
wrote:
On 29 Mar 2017, at 13:29, John Sanda wrote:
> however, deletion does not result in immediate deletion of data
> points. We still let data points in in the data_compressed table
> expire. We do this because one of the guidelines for using
> TimeWindowCompactionStrategy (TWCS) is to use a global TTL and not
> delete data._______________________________________________
This needs to be well documented as this is not what you
expect when you delete metrics in an urgency to free up space.
Yes, it does. The API docs mention this. It needs to be included in the user guide as
well. I have read that deleting data when using TWCS can actually cause it to hang around
longer than it would if you wait for it to expire, potentially resulting in more space
being used. If we really want to delete data vs just letting it expire, then we should not
be using TWCS, but I really think we want TWCS as it is designed specifically for time
series data. We are using TWCS with compressed data. A week’s worth of compressed data for
a metric is going to be quite small.
I suppose we need a section on deletes and how it works. The only ways in Cassandra to
immediately purge data are:
* drop table
* truncate table
* manually purge sstables
TWCS basically deletes an entire sstable when it determines it is safe to do so.
11:22 p.m.
The idea to limit inventory's responsibilities, and then the metrics TTL
discussion, sort of led me towards another idea. Perhaps we could
eliminate inventory and metrics from this feature, and also let the
agent be "dumb" and just configuration driven. We could add agent
meta-data on resource types we want reported when newly-discovered. I
suppose we could have a variety of notification-types but
"on-initial-discovery" would be the only thing of interest at the
moment, and only on server types.
The agent could make a rest call to hServices (glue code) notifying of a
new resource discovery. It could report this multiple times (like on
every startup). The glue code would just check to see if there was
already an event for this and if not it would generate one. This is via
the alerting API, bypassing metrics and inventory, no TTL issues, etc.
The logic is centralized and the agent doesn't really have to do
anything very smart (I think). We could feasibly do other things in the
hServices code, like update an existing event (via tag) each time to
keep track of recency, perhaps allowing some sort of event generation
for "removed resource" when we haven't been updated in a long time.
Thoughts?
On 3/29/2017 11:40 AM, Heiko W.Rupp wrote:
On 29 Mar 2017, at 15:22, John Sanda wrote:
>> This needs to be well documented as this is not what you
>> expect when you delete metrics in an urgency to free up space.
> [..] It needs to be included in the user guide as well.
This is what I meant. Sorry for not being more clear.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
12:15 a.m.
The idea to limit inventory's responsibilities, and then the
metrics TTL
discussion, sort of led me towards another idea. Perhaps we could
eliminate inventory and metrics from this feature, and also let the
agent be "dumb" and just configuration driven. We could add agent
meta-data on resource types we want reported when newly-discovered. I
suppose we could have a variety of notification-types but
"on-initial-discovery" would be the only thing of interest at the
moment, and only on server types.
The agent could make a rest call to hServices (glue code) notifying of a
new resource discovery. It could report this multiple times (like on
every startup). The glue code would just check to see if there was
already an event for this and if not it would generate one. This is via
the alerting API, bypassing metrics and inventory, no TTL issues, etc.
The logic is centralized and the agent doesn't really have to do
anything very smart (I think). We could feasibly do other things in the
hServices code, like update an existing event (via tag) each time to
keep track of recency, perhaps allowing some sort of event generation
for "removed resource" when we haven't been updated in a long time.
Thoughts?
To expand on the metadata idea - assuming we may want something else like this in the
future (maybe, "resource removed"??) we could have something on resource type
like "server-notification" and its a list of notifications we want the agent to
send up for resources of that type.
Say:
resource-type-dmr:
- name: My Server
- server-notifications:
- name: resource-added
- name: resource-removed
So when a resource of that type is added, the agent will send some "resource
added" event to the server. If a resource of that type is removed the agent will send
some "resource removed" event to the server (which we won't implement yet,
but that's the idea how we can add events in the future if we need them). The names of
the notifications are a fixed set. The agent would need to document the notification names
it supports. We would only support one to start - "resource-added" - for this
feature we are talking about in this thread.
The glue code can then take over and do what needs to be done for the messages the agent
sends.
We would need at least one REST endpoint in the server. One per notification type, I
guess.
Note that this won't work in "Metrics Only" mode and would be disabled since
the glue code only exists in "hawkular full mode".
2:32 a.m.
Right, I think we're on the same page. I'm going to try and throw
together a server-side POC for review.
On 4/3/2017 11:15 AM, John Mazzitelli wrote:
> The idea to limit inventory's responsibilities, and then the
metrics TTL
> discussion, sort of led me towards another idea. Perhaps we could
> eliminate inventory and metrics from this feature, and also let the
> agent be "dumb" and just configuration driven. We could add agent
> meta-data on resource types we want reported when newly-discovered. I
> suppose we could have a variety of notification-types but
> "on-initial-discovery" would be the only thing of interest at the
> moment, and only on server types.
>
> The agent could make a rest call to hServices (glue code) notifying of a
> new resource discovery. It could report this multiple times (like on
> every startup). The glue code would just check to see if there was
> already an event for this and if not it would generate one. This is via
> the alerting API, bypassing metrics and inventory, no TTL issues, etc.
> The logic is centralized and the agent doesn't really have to do
> anything very smart (I think). We could feasibly do other things in the
> hServices code, like update an existing event (via tag) each time to
> keep track of recency, perhaps allowing some sort of event generation
> for "removed resource" when we haven't been updated in a long time.
>
> Thoughts?
To expand on the metadata idea - assuming we may want something else like this in the
future (maybe, "resource removed"??) we could have something on resource type
like "server-notification" and its a list of notifications we want the agent to
send up for resources of that type.
Say:
resource-type-dmr:
- name: My Server
- server-notifications:
- name: resource-added
- name: resource-removed
So when a resource of that type is added, the agent will send some "resource
added" event to the server. If a resource of that type is removed the agent will send
some "resource removed" event to the server (which we won't implement yet,
but that's the idea how we can add events in the future if we need them). The names of
the notifications are a fixed set. The agent would need to document the notification names
it supports. We would only support one to start - "resource-added" - for this
feature we are talking about in this thread.
The glue code can then take over and do what needs to be done for the messages the agent
sends.
We would need at least one REST endpoint in the server. One per notification type, I
guess.
Note that this won't work in "Metrics Only" mode and would be disabled
since the glue code only exists in "hawkular full mode".
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
6:25 a.m.
I have a PR https://github.com/hawkular/hawkular-services/pull/136 for
the server-side POC. Anyone interested can take a look. Minimally,
Mazz and Heiko, please add any comments you may have.
On 4/3/2017 1:32 PM, Jay Shaughnessy wrote:
Right, I think we're on the same page. I'm going to try and throw
together a server-side POC for review.
On 4/3/2017 11:15 AM, John Mazzitelli wrote:
>> The idea to limit inventory's responsibilities, and then the metrics TTL
>> discussion, sort of led me towards another idea. Perhaps we could
>> eliminate inventory and metrics from this feature, and also let the
>> agent be "dumb" and just configuration driven. We could add agent
>> meta-data on resource types we want reported when newly-discovered. I
>> suppose we could have a variety of notification-types but
>> "on-initial-discovery" would be the only thing of interest at the
>> moment, and only on server types.
>>
>> The agent could make a rest call to hServices (glue code) notifying of a
>> new resource discovery. It could report this multiple times (like on
>> every startup). The glue code would just check to see if there was
>> already an event for this and if not it would generate one. This is via
>> the alerting API, bypassing metrics and inventory, no TTL issues, etc.
>> The logic is centralized and the agent doesn't really have to do
>> anything very smart (I think). We could feasibly do other things in the
>> hServices code, like update an existing event (via tag) each time to
>> keep track of recency, perhaps allowing some sort of event generation
>> for "removed resource" when we haven't been updated in a long
time.
>>
>> Thoughts?
> To expand on the metadata idea - assuming we may want something else like this in the
future (maybe, "resource removed"??) we could have something on resource type
like "server-notification" and its a list of notifications we want the agent to
send up for resources of that type.
>
> Say:
>
> resource-type-dmr:
> - name: My Server
> - server-notifications:
> - name: resource-added
> - name: resource-removed
>
> So when a resource of that type is added, the agent will send some "resource
added" event to the server. If a resource of that type is removed the agent will send
some "resource removed" event to the server (which we won't implement yet,
but that's the idea how we can add events in the future if we need them). The names of
the notifications are a fixed set. The agent would need to document the notification names
it supports. We would only support one to start - "resource-added" - for this
feature we are talking about in this thread.
>
> The glue code can then take over and do what needs to be done for the messages the
agent sends.
>
> We would need at least one REST endpoint in the server. One per notification type, I
guess.
>
> Note that this won't work in "Metrics Only" mode and would be disabled
since the glue code only exists in "hawkular full mode".
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2590
days inactive
2598
days old
16 comments
6 participants
participants (6)
-
Heiko W.Rupp -
Jay Shaughnessy -
Joel Takvorian -
John Mazzitelli -
John Sanda -
Stefan Negrea