agent "new server" event
by John Mazzitelli
<tl;dr>
Need ideas on how we are to implement the following two things in the agent:
1. At startup, agent needs to ask H-Metrics "what top-level servers have I told you about in an earlier life?"
2. When a new server is discovered, the agent should send an event to the server about the new server EXCEPT if the server isn't really new at all (see 1. above)
</tl;dr>
===
This post is to open up a discussion on how we want to implement a new features in the agent.
Joel is developing a new "inventory in metrics" feature: https://github.com/hawkular/hawkular-agent/pull/303
This means the agent will be storing inventory directly into Hawkular-Metrics. Because of this, we need to figure out how to get events sent based on things happening in H-Metric's inventory so MiQ can do things with it (like put things in the timeline such as "new server discovered" or "new WAR was deployed").
Jay looked at the code and the only thing that would be "missing" after this move of inventory into metrics is an event triggered when a new server is added to inventory. (When a new deployment is added, or a deployment is removed, the server is looking at command responses and generating events from that - so we don't lose anything by moving inventory into metrics).
By "new server", what we mean is a new resource that has no parent resources (i.e. a "root resource"). This includes standalone WildFly Servers and domain Host Controllers.
Right now, the agent starts with a "clean slate" when it starts up for the first time, or restarts. That means the agent's in-memory inventory graph is completely empty at startup - when discovery is run, the agent's internal inventory graph is filled in. After that, the agent just keeps the inventory graph up to date as it discovers new things coming and old things going away.
We need the agent to know if it already stored its top level servers into H-Metrics inventory and if it did, not to generate any "new server event". But if the agent is brand new, and it never sent any top-level resources to H-Metrics inventory yet, it should now send a "new server" event to the server (the agent never sent events like this before).
So there are two new things (assuming we keep the stuff Joel is doing - that is, we store inventory into H-Metrics):
1. At startup, agent needs to ask H-Metrics "what top-level servers have I told you about in an earlier life?"
2. When a new server is discovered, the agent should send an event (whatever this means - probably a REST API call somewhere) about the new server EXCEPT if the server isn't really new at all (see 1. above)
We need to figure out how to implement 1. and 2. So we are soliciting thoughts on those two subjects.
8 years, 9 months
Re: [Hawkular-dev] agent "new server" event
by Joel Takvorian
It could be interesting to have the possibility to deactivate TTL (for
instance by setting a negative value, without any change in the existing
API for that) but for the time being we could have the workaround of
setting an arbitrary high value, no?
Concerning the time range, at some point I was using "fromEarliest: true",
" order: desc" and "limit:1" ... It seems that it could also be used here.
Joel
Le 28 mars 2017 08:18, "John Sanda" <jsanda(a)redhat.com> a écrit :
My first thought was a string metric where data points are the servers that
get discovered. There are a couple things though that I do not like about
this. First, all data point queries in hawkular-metrics have a date range.
Having to use a date range here seems a bit awkward. Secondly all data
points in hawkular-metrics expire. These does not seem like data that we
would want to expire.
On Mar 27, 2017, at 5:02 PM, Jay Shaughnessy <jshaughn(a)redhat.com> wrote:
i was thinking, perhaps it's not even necessary to involve inventory to
know if the server was reported or not. We define metrics, like avail, for
these root types, I think. At startup if the metric existed perhaps you
could assume it was already reported, otherwise you could send a "new
server" event. Would that approach fly or be easier?
On 3/27/2017 12:40 PM, Joel Takvorian wrote:
For point 1., we can probably use some functions I wrote in the integration
tests, see there: https://github.com/jotak/hawkular-agent/blob/
inventory-strings/hawkular-agent-itest-util/src/main/
java/org/hawkular/agent/itest/util/ITestHelper.java
I assume you can build a canonical path? (The same "canonical path" than in
the existing inventory) If so, the method "getBlueprintFromCP" gives it to
you as an Optional blueprint.
On Mon, Mar 27, 2017 at 5:00 PM, John Mazzitelli <mazz(a)redhat.com> wrote:
> <tl;dr>
>
> Need ideas on how we are to implement the following two things in the
> agent:
>
> 1. At startup, agent needs to ask H-Metrics "what top-level servers have I
> told you about in an earlier life?"
>
> 2. When a new server is discovered, the agent should send an event to the
> server about the new server EXCEPT if the server isn't really new at all
> (see 1. above)
>
> </tl;dr>
>
>
> ===
>
>
> This post is to open up a discussion on how we want to implement a new
> features in the agent.
>
> Joel is developing a new "inventory in metrics" feature:
> https://github.com/hawkular/hawkular-agent/pull/303
>
> This means the agent will be storing inventory directly into
> Hawkular-Metrics. Because of this, we need to figure out how to get events
> sent based on things happening in H-Metric's inventory so MiQ can do things
> with it (like put things in the timeline such as "new server discovered" or
> "new WAR was deployed").
>
> Jay looked at the code and the only thing that would be "missing" after
> this move of inventory into metrics is an event triggered when a new server
> is added to inventory. (When a new deployment is added, or a deployment is
> removed, the server is looking at command responses and generating events
> from that - so we don't lose anything by moving inventory into metrics).
>
> By "new server", what we mean is a new resource that has no parent
> resources (i.e. a "root resource"). This includes standalone WildFly
> Servers and domain Host Controllers.
>
> Right now, the agent starts with a "clean slate" when it starts up for the
> first time, or restarts. That means the agent's in-memory inventory graph
> is completely empty at startup - when discovery is run, the agent's
> internal inventory graph is filled in. After that, the agent just keeps the
> inventory graph up to date as it discovers new things coming and old things
> going away.
>
> We need the agent to know if it already stored its top level servers into
> H-Metrics inventory and if it did, not to generate any "new server event".
> But if the agent is brand new, and it never sent any top-level resources to
> H-Metrics inventory yet, it should now send a "new server" event to the
> server (the agent never sent events like this before).
>
> So there are two new things (assuming we keep the stuff Joel is doing -
> that is, we store inventory into H-Metrics):
>
> 1. At startup, agent needs to ask H-Metrics "what top-level servers have I
> told you about in an earlier life?"
>
> 2. When a new server is discovered, the agent should send an event
> (whatever this means - probably a REST API call somewhere) about the new
> server EXCEPT if the server isn't really new at all (see 1. above)
>
> We need to figure out how to implement 1. and 2. So we are soliciting
> thoughts on those two subjects.
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
_______________________________________________
hawkular-dev mailing
listhawkular-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
8 years, 9 months
hawkular java agent now supports ${x} in config
by John Mazzitelli
Just released Hawkular Java Agent 0.29.1.Final that contains a new feature.
To support deploying Java Agent within containers (though this feature isn't restricted to that use-case), you can configure many settings in the Java Agent YAML config with ${x} expressions (similar to how HWFA can be configured in standalone.xml).
It supports getting values from system properties, environment variables, with default values optionally defined. The following expressions are supported - hopefully this is self-explanatory:
${some.system.property}
${some.system.property:a-default-value}
${ENV~SOME_ENV_VAR}
${ENV~SOME_ENV_VAR:a-default-value}
Not all properties support expressions (most don't need them) but properties that define things like hosts, ports, urls, passwords, filepaths, do support expressions.
If I missed any, or if you'd like a property to support expressions that does not currently, write a HWKAGENT JIRA: https://issues.jboss.org/projects/HWKAGENT
8 years, 9 months
hawkular java agent to be merged / released
by John Mazzitelli
FYI: I am going to merge the PR that introduces the javaagent and then I will release a new agent from that new master branch.
Just to be clear, here is what will then be supported:
1) The Hawkular WildFly Agent (HWFA) remains as-is. Nothing changes. The agent you've come to love and enjoy doesn't change.
2) There is a new Java Agent (HJA) that you can run in any JVM (including but not limited to EAP-based projects). You do so by passing in a "-javaagent" command line option to your JVM (e.g. java -javaagent=hawkular-javaagent.jar=config=config.yaml -jar my-app.jar ...yadda...). There are two additional properties you must set in standalone.conf if you want to run it inside an EAP JVM. The README will have the details.
This new HJA is configured with a yaml file that largely mimics all the standalone.xml data that HWFA has. There is no ${x} support in the YAML file right now.
This new HJA can talk to any EAP or WildFly server over the DMR management API. It can do deployments to your EAP/WildFly servers and monitoring of EAP/WildFly subsystems just like HWFA can.
This new HJA can talk to any JMX server just like HWFA can (it will talk to the local MBeanServer or, if remotely monitoring a JMX server, it will talk to it over Jolokia/REST API).
This new HJA can NOT run directly inside of a Host Controller due to https://issues.jboss.org/browse/WFCORE-2526 - however, you can run HJA externally in its own JVM (e.g. java -jar hawkular-javaagent.jar config=config.yaml) and have its config.yaml point to a remote Host Controller and you'll get the same functionality.
--John Mazz
8 years, 10 months
Garbage collection of outdated Servers/Metrics - especially with (orchestrated) containers
by Heiko W.Rupp
Hi,
I know I brought that up in the past ... (and I am sure Juca will easily
find where :-)
In the good old datacenter, one deployed an application server with
application, added it to monitoring and it hummed there happily. The
server sometimes got rebooted for OS updates or other things, but the
app-server always stayed the same and the monitoring system knew all its
pets by their name and the admins happily did this:
http://www.starwars-union.de/bilder/news/20110401_sunset.jpg
Nowadays in orchestration system the situation more looks like this
http://www.animationsfilme.ch/wp-content/uploads/2013/07/DespicableMe_01.jpg
where containers come and go and an application in container once it has
died is not re-started but a new container with its own ID is started.
Of course we can identify applications with labels so that I don't need
to know the container id. So I can gather and display metrics for those
and all is fine.
But: all those containers will create new
- metric ids
- inventory entries
- ???
The question is now: how long do we want/need to keep them?
Hawkular-metrics has a TTL for the datapoints, but I think metric
definitions are not evicted.
Similar a container being killed can't easily tell inventory that its
entry can go away.
For inventory-new we could use the expiration feature of
Hawkular-metrics for datapoints, where e.g. the agent would regularly
sync data and thus refresh the last-seen time to keep an entry "alive".
Also for the pure metrics - how much of historic data do we want/need.
And if we would e.g. aggregate those for long(er) term storage I think
we could perhaps actually aggregate over all individual time series over
many parallel pods and aggregate them into one for the entire
application.
Heiko
8 years, 10 months
hawkular java agent ready - itests passing
by John Mazzitelli
OK, I have been feverishly finishing this up. I believe this PR is ready. We now have a java agent that can be used in the place of the agent subsystem.
https://github.com/hawkular/hawkular-agent/pull/302
All the unit tests and itests pass - I even have itests running against the javaagent so I can say its working at least as well as the agent as a subsystem. The itests are essentially a copy of the ones we have for the wildfly subsystem agent so I'm exercising the javaagent in the same ways.
Two caveats that I can think of:
1) you can't run javaagent inside host controller directly. You have to run it as a separate process (the javaagent can also run on in its own JVM via "java -jar hawkular-javaagent.jar - it does not HAVE to be attached to some other JVM) and point to the host controller remotely. If you want to run it inside Host Controller, use the Hawkular WildFly Agent as a subsystem (i.e. use it like you do now with the original agent).
2) The "local" DMR mode isn't really local - under the covers it is going through the remote management endpoint so long as the WildFly EAP is configured on the default 127.0.0.1:9990. If it is not, just configure the agent to talk to its local WildFly as if it were a remote-dmr managed server.
To solve 1 we need the host controller to incorporate this enhancement: https://issues.jboss.org/browse/WFCORE-2526
To solve 2 I need to play games with classloading and JBoss Modules - I was given some ideas from some folks. I plan to try something to get that to work.
This javaagent has JMX support. It talks local JMX API if local, and Jolokia REST if remote. It supports cmdgw websocket command ExecuteOperationRequest to invoke JMX operations (itests show this working). So we have full DMR and JMX support.
So this is ready for people to try out for non-EAP based products with JMX as their management interface. Tom Cunningham is testing now. He already spotted a packaging issue with sprintboot which he is figuring out - I'll let him explain if he has the energy :)
I think I can merge this soon, unless someone has reservations.
FWIW: the wildfly agent subsystem support remains intact and as it was. So that didn't change and works like it always has (its itests are still there and passing so I know I didn't break those :).
8 years, 10 months
Inventory storage on metrics
by Joel Takvorian
Quick reminder for those who may not have followed the discussions: this is
about removing the inventory project as it currently stands, and replace it
with Metrics-based (so C*-based) storage, using String metrics and json
blobs.
So, I've been analyzing the usages of inventory in the wildfly agent and in
manageiq / ruby client.
As posted on another topic, I started with the following pattern for
putting inventories in string metrics:
- metrics are named:
*inventory.[feedid].[r|rt|mt].[id]*
which means that we would store JSON blobs by resource type, by metric type
and by root resource (all per feed id). Which also means we don't allow
anything outside of a feed.
This naming pattern seems to be quite easily transposable to the wildfly
agent feeding.
Things become more complicated when we look at the ruby client / manageIQ.
To help in querying, we can add those tags on every metric:
module: inventory
feed: [feedid]
type: [r|rt|mt]
With this, we can easily query, for instance, all resource types of a given
feed, or all feeds, or child resources at a given path, etc.
Now, after analysis of the ruby code (and I hope I didn't miss important
stuff), I have two remarks:
- There is a lot a methods in the ruby client that are not used in manage
IQ (listed below [1]). Of course those methods may have an interest for a
general purpose client, but since we're removing a lot of functionalities
in inventory, I would suggest not to focus on them, and just remove them
for the time being. This is possible, of course, if no other product than
manageIQ consumes then... is it the case?
- I've found just 1 method in the ruby client that would be badly impacted
by the usage of string metrics as currently defined. And unfortunately it's
also the one that is most called (in number of occurrence in code) by
manage IQ, it's "list_resources_for_type"; With the implementation I
described above, to list all resources of a given type we would have to get
all resource blobs of a feed and search in all of them to check all root &
child resource types. That would probably be terrible for performances.
A suggestion to this problem, is to tag each root resource with one tag per
resource type, something like: "
*rt.MyResourceType=resource1,resource2,resource3*".
Then, we would still have to load the whole blobs and search resources in
them, but we would exclude from the search all root resources that doesn't
have those types, and we wouldn't have to search extensively in the blobs
as we would know the paths.
*Question to metrics devs* about that: can we use any character for
tagging, or is there some limitation?
Or if you can suggest a better way, I'm all ears :)
[1] list of methods that seem to be unused from the ruby client:
- list_resources_for_feed
- list_relationships
- list_relationships_for_feed
- get_entity
- list_metrics_for_metric_type
- list_metrics_for_resource_type
- create_feed
- delete_feed
- create_resource_type
- create_resource
- create_resource_under_resource
- create_metric_type
- list_operation_definitions
- list_operation_definitions_for_resource
- create_metric_for_resource
- no_more_events
Joel
8 years, 10 months
