From jshaughn at redhat.com Mon Apr 3 10:22:34 2017 From: jshaughn at redhat.com (Jay Shaughnessy) Date: Mon, 3 Apr 2017 10:22:34 -0400 Subject: [Hawkular-dev] agent "new server" event In-Reply-To: References: <885676728.15015672.1490625609860.JavaMail.zimbra@redhat.com> <148032350.15045295.1490626857364.JavaMail.zimbra@redhat.com> <0f7ebbf7-f52b-8c69-2125-3523f411b806@redhat.com> <209DD9E0-C655-494F-AD87-0EF94F9D024F@redhat.com> <95AF21FF-CD99-4EAB-AA45-73D91C46852C@redhat.com> Message-ID: The idea to limit inventory's responsibilities, and then the metrics TTL discussion, sort of led me towards another idea. Perhaps we could eliminate inventory and metrics from this feature, and also let the agent be "dumb" and just configuration driven. We could add agent meta-data on resource types we want reported when newly-discovered. I suppose we could have a variety of notification-types but "on-initial-discovery" would be the only thing of interest at the moment, and only on server types. The agent could make a rest call to hServices (glue code) notifying of a new resource discovery. It could report this multiple times (like on every startup). The glue code would just check to see if there was already an event for this and if not it would generate one. This is via the alerting API, bypassing metrics and inventory, no TTL issues, etc. The logic is centralized and the agent doesn't really have to do anything very smart (I think). We could feasibly do other things in the hServices code, like update an existing event (via tag) each time to keep track of recency, perhaps allowing some sort of event generation for "removed resource" when we haven't been updated in a long time. Thoughts? On 3/29/2017 11:40 AM, Heiko W.Rupp wrote: > On 29 Mar 2017, at 15:22, John Sanda wrote: > >>> This needs to be well documented as this is not what you >>> expect when you delete metrics in an urgency to free up space. >> [..] It needs to be included in the user guide as well. > This is what I meant. Sorry for not being more clear. > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > From mazz at redhat.com Mon Apr 3 11:15:22 2017 From: mazz at redhat.com (John Mazzitelli) Date: Mon, 3 Apr 2017 11:15:22 -0400 (EDT) Subject: [Hawkular-dev] agent "new server" event In-Reply-To: References: <885676728.15015672.1490625609860.JavaMail.zimbra@redhat.com> <209DD9E0-C655-494F-AD87-0EF94F9D024F@redhat.com> <95AF21FF-CD99-4EAB-AA45-73D91C46852C@redhat.com> Message-ID: <223314016.19527373.1491232522901.JavaMail.zimbra@redhat.com> > The idea to limit inventory's responsibilities, and then the metrics TTL > discussion, sort of led me towards another idea. Perhaps we could > eliminate inventory and metrics from this feature, and also let the > agent be "dumb" and just configuration driven. We could add agent > meta-data on resource types we want reported when newly-discovered. I > suppose we could have a variety of notification-types but > "on-initial-discovery" would be the only thing of interest at the > moment, and only on server types. > > The agent could make a rest call to hServices (glue code) notifying of a > new resource discovery. It could report this multiple times (like on > every startup). The glue code would just check to see if there was > already an event for this and if not it would generate one. This is via > the alerting API, bypassing metrics and inventory, no TTL issues, etc. > The logic is centralized and the agent doesn't really have to do > anything very smart (I think). We could feasibly do other things in the > hServices code, like update an existing event (via tag) each time to > keep track of recency, perhaps allowing some sort of event generation > for "removed resource" when we haven't been updated in a long time. > > Thoughts? To expand on the metadata idea - assuming we may want something else like this in the future (maybe, "resource removed"??) we could have something on resource type like "server-notification" and its a list of notifications we want the agent to send up for resources of that type. Say: resource-type-dmr: - name: My Server - server-notifications: - name: resource-added - name: resource-removed So when a resource of that type is added, the agent will send some "resource added" event to the server. If a resource of that type is removed the agent will send some "resource removed" event to the server (which we won't implement yet, but that's the idea how we can add events in the future if we need them). The names of the notifications are a fixed set. The agent would need to document the notification names it supports. We would only support one to start - "resource-added" - for this feature we are talking about in this thread. The glue code can then take over and do what needs to be done for the messages the agent sends. We would need at least one REST endpoint in the server. One per notification type, I guess. Note that this won't work in "Metrics Only" mode and would be disabled since the glue code only exists in "hawkular full mode". From jshaughn at redhat.com Mon Apr 3 13:32:43 2017 From: jshaughn at redhat.com (Jay Shaughnessy) Date: Mon, 3 Apr 2017 13:32:43 -0400 Subject: [Hawkular-dev] agent "new server" event In-Reply-To: <223314016.19527373.1491232522901.JavaMail.zimbra@redhat.com> References: <885676728.15015672.1490625609860.JavaMail.zimbra@redhat.com> <209DD9E0-C655-494F-AD87-0EF94F9D024F@redhat.com> <95AF21FF-CD99-4EAB-AA45-73D91C46852C@redhat.com> <223314016.19527373.1491232522901.JavaMail.zimbra@redhat.com> Message-ID: <1523bf41-fd7a-08ef-22e6-384b16ac5ff9@redhat.com> Right, I think we're on the same page. I'm going to try and throw together a server-side POC for review. On 4/3/2017 11:15 AM, John Mazzitelli wrote: >> The idea to limit inventory's responsibilities, and then the metrics TTL >> discussion, sort of led me towards another idea. Perhaps we could >> eliminate inventory and metrics from this feature, and also let the >> agent be "dumb" and just configuration driven. We could add agent >> meta-data on resource types we want reported when newly-discovered. I >> suppose we could have a variety of notification-types but >> "on-initial-discovery" would be the only thing of interest at the >> moment, and only on server types. >> >> The agent could make a rest call to hServices (glue code) notifying of a >> new resource discovery. It could report this multiple times (like on >> every startup). The glue code would just check to see if there was >> already an event for this and if not it would generate one. This is via >> the alerting API, bypassing metrics and inventory, no TTL issues, etc. >> The logic is centralized and the agent doesn't really have to do >> anything very smart (I think). We could feasibly do other things in the >> hServices code, like update an existing event (via tag) each time to >> keep track of recency, perhaps allowing some sort of event generation >> for "removed resource" when we haven't been updated in a long time. >> >> Thoughts? > To expand on the metadata idea - assuming we may want something else like this in the future (maybe, "resource removed"??) we could have something on resource type like "server-notification" and its a list of notifications we want the agent to send up for resources of that type. > > Say: > > resource-type-dmr: > - name: My Server > - server-notifications: > - name: resource-added > - name: resource-removed > > So when a resource of that type is added, the agent will send some "resource added" event to the server. If a resource of that type is removed the agent will send some "resource removed" event to the server (which we won't implement yet, but that's the idea how we can add events in the future if we need them). The names of the notifications are a fixed set. The agent would need to document the notification names it supports. We would only support one to start - "resource-added" - for this feature we are talking about in this thread. > > The glue code can then take over and do what needs to be done for the messages the agent sends. > > We would need at least one REST endpoint in the server. One per notification type, I guess. > > Note that this won't work in "Metrics Only" mode and would be disabled since the glue code only exists in "hawkular full mode". > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170403/3c85d773/attachment.html From theute at redhat.com Tue Apr 4 04:06:24 2017 From: theute at redhat.com (Thomas Heute) Date: Tue, 4 Apr 2017 10:06:24 +0200 Subject: [Hawkular-dev] Hawkular Alerting 1.6.0.Final is Available! In-Reply-To: <43f25842-428a-fa0f-ab30-be5622d5c831@redhat.com> References: <43f25842-428a-fa0f-ab30-be5622d5c831@redhat.com> Message-ID: Thank you and congrats ! On Fri, Mar 31, 2017 at 7:38 PM, Jay Shaughnessy wrote: > > The Hawkular Alerting team is happy to announce the release of Hawkular > Alerting 1.6.0.Final. This release introduces several new features as > well as some important fixes. > Features > > - Extension for Aggregated Events Alerting > - Sliding windows to aggregate events and perform robust condition > evaluations. > - See http://www.hawkular.org/community/docs/developer- > guide/alerts.html#_events_aggregation_extension > > - Tag Query Language > - Powerful tag search with a new tag query language. > - See http://www.hawkular.org/community/docs/developer- > guide/alerts.html#_tags > - Integration with Elasticsearch > - Extension for conditions that query Elasticsearch. > - Action Plugin for writing back to Elasticsearch, > - See http://www.hawkular.org/community/docs/developer- > guide/alerts.html#_elasticsearch_integration > > - Watcher Support > - Open a connection and stream new alerts or events filtered by > robust criteria > - See http://www.hawkular.org/docs/rest/rest-alerts.html#GET__watch > - Hawkular Metrics Integration > - New Metrics Group Trigger feature to track member triggers > automatically > - See http://www.hawkular.org/hawkular-metrics/docs/user- > guide/#_metrics_group_triggers > - New Tutorial > - Easy on-boarding with docker-based tutorial. First 5 lessons > available. > - See http://www.hawkular.org/blog/2017/03/14/hawkular-alerting- > tutorial.html > > Enhancements > > - REST Endpoints > - New endpoints for trigger enable/disable > - Set all group trigger conditions, regardless of their triggerMode > - Action TimeConstraints can now tie the constraint to a specific > TimeZone > - Support group members on Import. > > Fixes > > - [HWKALERTS-222 ] - > Cassalog schema updates are lacking verification functions > - [HWKALERTS-223 ] - > Propagate Listener events on cluster scenarios > - [HWKALERTS-227 ] - > Schema bootstrap skip should handle failures and server shutdowns > - [HWKALERTS-229 ] - > Setting the conditions for group trigger ignores the condition triggerMode > and follows only the triggerMode in the path parameters > - [HWKALERTS-234 ] - > MissingCondition handling needs to consider trigger mode > - [HWKALERTS-243 ] - > DefinitionsEvents might have a wrong order on import operations > - [HWKALERTS-249 ] - > Schema installation on multi Cassandra node cluster may fail > > > > *Hawkular Alerting 1.6.0.Final is available:* > > - Immediately as a standalone distribution > - Soon as part of Hawkular Metrics 0.26.0.Final > - In Hawkular Services for ManageIQ > - In OpenShift Online > > > For more details about this release: https://issues.jboss.org/ > secure/ReleaseNote.jspa?projectId=12315924&version=12333585 > > http://www.hawkular.org/ > http://www.hawkular.org/community/docs/developer-guide/alerts.html > http://www.hawkular.org/docs/rest/rest-alerts.html > > https://github.com/hawkular/hawkular-alerts > https://github.com/hawkular/hawkular-alerts/tree/master/examples > > #hawkular on freenode > > Hawkular Alerting Team > Jay Shaughnessy (jshaughn at redhat.com) > Lucas Ponce (lponce at redhat.com) > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170404/6ed7bb49/attachment-0001.html From jshaughn at redhat.com Tue Apr 4 17:25:22 2017 From: jshaughn at redhat.com (Jay Shaughnessy) Date: Tue, 4 Apr 2017 17:25:22 -0400 Subject: [Hawkular-dev] agent "new server" event In-Reply-To: <1523bf41-fd7a-08ef-22e6-384b16ac5ff9@redhat.com> References: <885676728.15015672.1490625609860.JavaMail.zimbra@redhat.com> <209DD9E0-C655-494F-AD87-0EF94F9D024F@redhat.com> <95AF21FF-CD99-4EAB-AA45-73D91C46852C@redhat.com> <223314016.19527373.1491232522901.JavaMail.zimbra@redhat.com> <1523bf41-fd7a-08ef-22e6-384b16ac5ff9@redhat.com> Message-ID: I have a PR https://github.com/hawkular/hawkular-services/pull/136 for the server-side POC. Anyone interested can take a look. Minimally, Mazz and Heiko, please add any comments you may have. On 4/3/2017 1:32 PM, Jay Shaughnessy wrote: > > Right, I think we're on the same page. I'm going to try and throw > together a server-side POC for review. > > On 4/3/2017 11:15 AM, John Mazzitelli wrote: >>> The idea to limit inventory's responsibilities, and then the metrics TTL >>> discussion, sort of led me towards another idea. Perhaps we could >>> eliminate inventory and metrics from this feature, and also let the >>> agent be "dumb" and just configuration driven. We could add agent >>> meta-data on resource types we want reported when newly-discovered. I >>> suppose we could have a variety of notification-types but >>> "on-initial-discovery" would be the only thing of interest at the >>> moment, and only on server types. >>> >>> The agent could make a rest call to hServices (glue code) notifying of a >>> new resource discovery. It could report this multiple times (like on >>> every startup). The glue code would just check to see if there was >>> already an event for this and if not it would generate one. This is via >>> the alerting API, bypassing metrics and inventory, no TTL issues, etc. >>> The logic is centralized and the agent doesn't really have to do >>> anything very smart (I think). We could feasibly do other things in the >>> hServices code, like update an existing event (via tag) each time to >>> keep track of recency, perhaps allowing some sort of event generation >>> for "removed resource" when we haven't been updated in a long time. >>> >>> Thoughts? >> To expand on the metadata idea - assuming we may want something else like this in the future (maybe, "resource removed"??) we could have something on resource type like "server-notification" and its a list of notifications we want the agent to send up for resources of that type. >> >> Say: >> >> resource-type-dmr: >> - name: My Server >> - server-notifications: >> - name: resource-added >> - name: resource-removed >> >> So when a resource of that type is added, the agent will send some "resource added" event to the server. If a resource of that type is removed the agent will send some "resource removed" event to the server (which we won't implement yet, but that's the idea how we can add events in the future if we need them). The names of the notifications are a fixed set. The agent would need to document the notification names it supports. We would only support one to start - "resource-added" - for this feature we are talking about in this thread. >> >> The glue code can then take over and do what needs to be done for the messages the agent sends. >> >> We would need at least one REST endpoint in the server. One per notification type, I guess. >> >> Note that this won't work in "Metrics Only" mode and would be disabled since the glue code only exists in "hawkular full mode". >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev >> > > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170404/63539c6f/attachment.html From mazz at redhat.com Tue Apr 4 22:13:55 2017 From: mazz at redhat.com (John Mazzitelli) Date: Tue, 4 Apr 2017 22:13:55 -0400 (EDT) Subject: [Hawkular-dev] agent release In-Reply-To: <466305135.21011795.1491358341527.JavaMail.zimbra@redhat.com> Message-ID: <1311896578.21011938.1491358435093.JavaMail.zimbra@redhat.com> Hawkular Agent release 0.29.4.Final is out. This includes both HWFA and HJA. The main reason for this release is to put out the feature that Tom Cunningham wanted - the ability for the agent to wait for certain resources to be available before the agent starts monitoring things. See https://issues.jboss.org/browse/HWKAGENT-165. This release also includes feature packs for HJA (EAP6 and EAP7/WF10+) and a distro that includes WF10+Java Agent (thanks josejulio for that stuff) From mazz at redhat.com Wed Apr 5 11:50:46 2017 From: mazz at redhat.com (John Mazzitelli) Date: Wed, 5 Apr 2017 11:50:46 -0400 (EDT) Subject: [Hawkular-dev] deploying HOSA without git cloning repo In-Reply-To: <1133562980.21640779.1491407037980.JavaMail.zimbra@redhat.com> Message-ID: <1281120492.21642165.1491407446479.JavaMail.zimbra@redhat.com> If you want to deploy HOSA, you need only this one command: wget https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/master/hack/deploy-openshift-agent.sh && sh ./deploy-openshift-agent.sh I had a few people ask how to deploy the Hawkular OpenShift Agent, but when I tried to explain it, it seemed harder than it should be :) For one, I don't want to ask people to git clone the source repo because you don't really need to (the docker image is on docker hub so you don't have to build anything). You really just need the commands the Makefile performs and you need a couple yaml files for the openshift templates. But even that gets too confusing ("download these two files, run these commands you see in the Makefile"). So I wrote a small script that does it for you. So if you want to deploy HOSA, just grab this script and run it. Should take care of everything: https://github.com/hawkular/hawkular-openshift-agent/blob/master/hack/deploy-openshift-agent.sh The comments at the top provide additional details. You can use wget to pull that file from github and then you can run it. So this command should be all you need to deploy an agent: wget https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/master/hack/deploy-openshift-agent.sh && sh ./deploy-openshift-agent.sh From mwringe at redhat.com Wed Apr 5 14:52:32 2017 From: mwringe at redhat.com (Matt Wringe) Date: Wed, 5 Apr 2017 14:52:32 -0400 (EDT) Subject: [Hawkular-dev] deploying HOSA without git cloning repo In-Reply-To: <1281120492.21642165.1491407446479.JavaMail.zimbra@redhat.com> References: <1281120492.21642165.1491407446479.JavaMail.zimbra@redhat.com> Message-ID: <949740382.11495834.1491418352385.JavaMail.zimbra@redhat.com> ----- Original Message ----- > From: "John Mazzitelli" > To: "Discussions around Hawkular development" > Sent: Wednesday, 5 April, 2017 11:50:46 AM > Subject: [Hawkular-dev] deploying HOSA without git cloning repo > > > If you want to deploy HOSA, you need only this one command: > wget > https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/master/hack/deploy-openshift-agent.sh > && sh ./deploy-openshift-agent.sh > > > I had a few people ask how to deploy the Hawkular OpenShift Agent, but when I > tried to explain it, it seemed harder than it should be :) Official Docs: https://docs.openshift.org/latest/install_config/cluster_metrics.html#deploying-hawkular-openshift-agent As handy as it is to use a bash script to deploy things, we don't in the docs for people who are not running on linux. > For one, I don't want to ask people to git clone the source repo because you > don't really need to (the docker image is on docker hub so you don't have to > build anything). You really just need the commands the Makefile performs and > you need a couple yaml files for the openshift templates. But even that gets > too confusing ("download these two files, run these commands you see in the > Makefile"). > > So I wrote a small script that does it for you. So if you want to deploy > HOSA, just grab this script and run it. Should take care of everything: > > https://github.com/hawkular/hawkular-openshift-agent/blob/master/hack/deploy-openshift-agent.sh > > The comments at the top provide additional details. > > You can use wget to pull that file from github and then you can run it. So > this command should be all you need to deploy an agent: > > wget > https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/master/hack/deploy-openshift-agent.sh > && sh ./deploy-openshift-agent.sh > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > From mazz at redhat.com Wed Apr 5 16:34:15 2017 From: mazz at redhat.com (John Mazzitelli) Date: Wed, 5 Apr 2017 16:34:15 -0400 (EDT) Subject: [Hawkular-dev] deploying HOSA without git cloning repo In-Reply-To: <949740382.11495834.1491418352385.JavaMail.zimbra@redhat.com> References: <1281120492.21642165.1491407446479.JavaMail.zimbra@redhat.com> <949740382.11495834.1491418352385.JavaMail.zimbra@redhat.com> Message-ID: <2075530101.21783066.1491424455058.JavaMail.zimbra@redhat.com> > > I had a few people ask how to deploy the Hawkular OpenShift Agent, but when > > I tried to explain it, it seemed harder than it should be :) > > Official Docs: > https://docs.openshift.org/latest/install_config/cluster_metrics.html#deploying-hawkular-openshift-agent > > As handy as it is to use a bash script to deploy things, we don't in the docs > for people who are not running on linux. Yeah. I just created this hack/ script to make it easy for most people. At least good enough to get linux users up and running fairly painlessly and quickly. BTW: I just noticed that those OpenShift docs are incorrect - its pointing to the wrong github links to wget. It says: === $ wget https://github.com/openshift/origin-metrics/blob/master/hawkular-agent/hawkular-openshift-agent-configmap.yaml $ wget https://github.com/openshift/origin-metrics/blob/master/hawkular-agent/hawkular-openshift-agent.yaml === But when you wget those, you are getting the actual HTML page for those files. We want to tell the user to get the raw files, whose links are: https://raw.githubusercontent.com/openshift/origin-metrics/master/hawkular-agent/hawkular-openshift-agent-configmap.yaml https://raw.githubusercontent.com/openshift/origin-metrics/master/hawkular-agent/hawkular-openshift-agent.yaml From hrupp at redhat.com Thu Apr 6 12:02:11 2017 From: hrupp at redhat.com (Heiko W.Rupp) Date: Thu, 06 Apr 2017 18:02:11 +0200 Subject: [Hawkular-dev] Hawkular web site and tags In-Reply-To: <8095533A-B14A-4367-BB2D-F3421E8A89D8@redhat.com> References: <8095533A-B14A-4367-BB2D-F3421E8A89D8@redhat.com> Message-ID: On 24 Mar 2017, at 12:10, Heiko W.Rupp wrote: Recently I made a change to the web site to enable tags, and have a "tag cloud". The tag cloud is now also at the bottom of http://www.hawkular.org Please make sure when writing blog post and/or pages to (re) use those. Tags with only one page/article don't really add value. But don't leave them out when you write about something completely new. Example "Apache camel" should of course be mentioned the 1st time someone writes about Camel integration. As you see with Apache Camel, it is possible to create multi word tags https://github.com/hawkular/hawkular.github.io/blame/pages/src/main/jbake/content/blog/2017/03/24/distributed-tracing-with-camel.adoc#L6 they fail to correctly link when running jbake locally, but work fine on GitHub pages. Also for e.g. distributed tracing the multi word tag "distributed tracing" is better than distributed, tracing. also please try to consistently use the same spelling and upper/lower case for a tag. apm != APM -- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht M?nchen HRB 153243 Gesch?ftsf?hrer: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander From hrupp at redhat.com Tue Apr 11 02:32:39 2017 From: hrupp at redhat.com (Heiko W.Rupp) Date: Tue, 11 Apr 2017 08:32:39 +0200 Subject: [Hawkular-dev] Article about Prometheus v2 storage format Message-ID: <25BDF81F-096A-40FE-9662-DD6840054387@redhat.com> A discussion about how P8s came to the storage format they use in P8s V2 https://fabxc.org/blog/2017-04-10-writing-a-tsdb/ From miburman at redhat.com Tue Apr 11 07:27:26 2017 From: miburman at redhat.com (Michael Burman) Date: Tue, 11 Apr 2017 14:27:26 +0300 Subject: [Hawkular-dev] Article about Prometheus v2 storage format In-Reply-To: <25BDF81F-096A-40FE-9662-DD6840054387@redhat.com> References: <25BDF81F-096A-40FE-9662-DD6840054387@redhat.com> Message-ID: Hi, What's funny is that in the end they reinvented yet-another-LSM and inverted index. Basically Cassandra (with SASI) without any scalability. - Micke On 04/11/2017 09:32 AM, Heiko W.Rupp wrote: > A discussion about how P8s came to the storage format they use in P8s V2 > > https://fabxc.org/blog/2017-04-10-writing-a-tsdb/ > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev From kurrent93 at gmail.com Tue Apr 11 12:17:01 2017 From: kurrent93 at gmail.com (Anton) Date: Tue, 11 Apr 2017 18:17:01 +0200 Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. Message-ID: Hello I have tried to run Hawkular in both `oc cluster up` and OpenShift, different versions of both. The documentation for `oc cluster up` says ( https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md#installing-metrics) that by using the `--metrics` parameter, metrics is installed. Its a similar story when installing OpenShift using Openshift-ansible. However, in both cases, when logging into the OpenShift webconsole I see an error message saying: An error occurred getting metrics. Open Metrics URL | Don't Show Me Again It would seem there is more to correctly installing Hawkular metrics than just using the parameter. What else needs to be done? And is it possible get the additional info added to the documentation for openshift-ansible and `oc cluster up`? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170411/f869cf08/attachment.html From mwringe at redhat.com Tue Apr 11 13:51:45 2017 From: mwringe at redhat.com (Matt Wringe) Date: Tue, 11 Apr 2017 13:51:45 -0400 (EDT) Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. In-Reply-To: References: Message-ID: <835243741.14041969.1491933105472.JavaMail.zimbra@redhat.com> ----- Original Message ----- > From: "Anton" > To: "Discussions" > Sent: Tuesday, 11 April, 2017 12:17:01 PM > Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. > > Hello > > I have tried to run Hawkular in both `oc cluster up` and OpenShift, different > versions of both. > > The documentation for `oc cluster up` says ( > https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md#installing-metrics > ) that by using the `--metrics` parameter, metrics is installed. > > Its a similar story when installing OpenShift using Openshift-ansible. > > However, in both cases, when logging into the OpenShift webconsole I see an > error message saying: > > An error occurred getting metrics. Open Metrics URL | Don't Show Me Again > What happens if you open the metrics URL? By default the metrics certificate is self signed and not trusted by the browser. So you will first need to click on the link and verify if you trust the certificate or not. If you accept the certificate and refresh the metrics page, do you still see the error? If not, you might be running into an issue. If that is the case, can you please attach the output of 'oc get pods -n openshift-infra' to see what state each of the pods are in? > It would seem there is more to correctly installing Hawkular metrics than > just using the parameter. > > What else needs to be done? And is it possible get the additional info added > to the documentation for openshift-ansible and `oc cluster up`? > > Thanks > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > From mazz at redhat.com Tue Apr 11 14:55:39 2017 From: mazz at redhat.com (John Mazzitelli) Date: Tue, 11 Apr 2017 14:55:39 -0400 (EDT) Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. In-Reply-To: References: Message-ID: <2020957409.25663719.1491936939322.JavaMail.zimbra@redhat.com> This happens always - and I never quite figured out why. But once you click on that link "Open Metrics URL" and get the "started" splash page, it all shows it is working. I think it has to do with the self-signed certificate and how the browser wants to be told it is OK. I'm not sure, that's just a guess. But, if you started "oc cluster up --metrics" and you see that message, its all OK. Just click "Open Metrics URL" and see that you can get to the metrics splash screen via your browser. Other than that, I don't have a good explanation - I just know how I get that message to go away. It shows up for me everytime I "oc cluster up --metrics". ----- Original Message ----- > Hello > > I have tried to run Hawkular in both `oc cluster up` and OpenShift, different > versions of both. > > The documentation for `oc cluster up` says ( > https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md#installing-metrics > ) that by using the `--metrics` parameter, metrics is installed. > > Its a similar story when installing OpenShift using Openshift-ansible. > > However, in both cases, when logging into the OpenShift webconsole I see an > error message saying: > > An error occurred getting metrics. Open Metrics URL | Don't Show Me Again > > > It would seem there is more to correctly installing Hawkular metrics than > just using the parameter. > > What else needs to be done? And is it possible get the additional info added > to the documentation for openshift-ansible and `oc cluster up`? > > Thanks > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > From kurrent93 at gmail.com Tue Apr 11 15:35:04 2017 From: kurrent93 at gmail.com (Anton) Date: Tue, 11 Apr 2017 21:35:04 +0200 Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. In-Reply-To: <835243741.14041969.1491933105472.JavaMail.zimbra@redhat.com> References: <835243741.14041969.1491933105472.JavaMail.zimbra@redhat.com> Message-ID: Hi Matt On 11 April 2017 at 19:51, Matt Wringe wrote: > > What happens if you open the metrics URL? > > By default the metrics certificate is self signed and not trusted by the > browser. So you will first need to click on the link and verify if you > trust the certificate or not. If you accept the certificate and refresh the > metrics page, do you still see the error? > Its a dns issue, rather than a cert issue: This site can?t be reached *hawkular-metrics.apps.console.mydomain.com *?s server DNS addresscould not be found. DNS_PROBE_FINISHED_NXDOMAIN > > If not, you might be running into an issue. If that is the case, can you > please attach the output of 'oc get pods -n openshift-infra' to see what > state each of the pods are in? # oc get pods -n openshift-infra NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-qnx2x 1/1 Running 0 3d hawkular-metrics-3qhpj 1/1 Running 5 3d heapster-69qzn 1/1 Running 4 3d -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170411/988d9591/attachment.html From kurrent93 at gmail.com Tue Apr 11 15:47:29 2017 From: kurrent93 at gmail.com (Anton) Date: Tue, 11 Apr 2017 21:47:29 +0200 Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. In-Reply-To: References: <835243741.14041969.1491933105472.JavaMail.zimbra@redhat.com> Message-ID: I created a new route, and this time I get a different two different errors. *In http:* 502 Bad GatewayThe server returned an invalid or incomplete response. *In https:* Application is not available The application is currently not serving requests at this endpoint. It may not have been started or is still starting. Possible reasons you are seeing this page: - *The host doesn't exist.* Make sure the hostname was typed correctly and that a route matching this hostname exists. - *The host exists, but doesn't have a matching path.* Check if the URL path was typed correctly and that the route was created using the desired path. - *Route and path matches, but all pods are down.* Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running. On 11 April 2017 at 21:35, Anton wrote: > Hi Matt > On 11 April 2017 at 19:51, Matt Wringe wrote: > >> >> What happens if you open the metrics URL? >> >> By default the metrics certificate is self signed and not trusted by the >> browser. So you will first need to click on the link and verify if you >> trust the certificate or not. If you accept the certificate and refresh the >> metrics page, do you still see the error? >> > > Its a dns issue, rather than a cert issue: > > This site can?t be reached > > *hawkular-metrics.apps.console.mydomain.com > *?s server DNS addresscould > not be found. > DNS_PROBE_FINISHED_NXDOMAIN > > >> >> If not, you might be running into an issue. If that is the case, can you >> please attach the output of 'oc get pods -n openshift-infra' to see what >> state each of the pods are in? > > > # oc get pods -n openshift-infra > NAME READY STATUS RESTARTS AGE > hawkular-cassandra-1-qnx2x 1/1 Running 0 3d > hawkular-metrics-3qhpj 1/1 Running 5 3d > heapster-69qzn 1/1 Running 4 3d > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170411/083a7d29/attachment-0001.html From mazz at redhat.com Tue Apr 11 16:03:20 2017 From: mazz at redhat.com (John Mazzitelli) Date: Tue, 11 Apr 2017 16:03:20 -0400 (EDT) Subject: [Hawkular-dev] test In-Reply-To: <380416206.25685194.1491940983799.JavaMail.zimbra@redhat.com> Message-ID: <252221930.25685482.1491941000238.JavaMail.zimbra@redhat.com> I'm sending this on April 11, 2017 at 4:03 PM EDT Let's see how long it takes to get posted. From kurrent93 at gmail.com Tue Apr 11 17:12:29 2017 From: kurrent93 at gmail.com (Anton) Date: Tue, 11 Apr 2017 23:12:29 +0200 Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. In-Reply-To: <2020957409.25663719.1491936939322.JavaMail.zimbra@redhat.com> References: <2020957409.25663719.1491936939322.JavaMail.zimbra@redhat.com> Message-ID: Hi John To clarify, its not a certificate issue. After I accept the certificate warning, the browser redirects to: Application is not available The application is currently not serving requests at this endpoint. It may not have been started or is still starting. Possible reasons you are seeing this page: The host doesn't exist. Make sure the hostname was typed correctly and that a route matching this hostname exists. The host exists, but doesn't have a matching path. Check if the URL path was typed correctly and that the route was created using the desired path. Route and path matches, but all pods are down. Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running. On 11 April 2017 at 20:55, John Mazzitelli wrote: > This happens always - and I never quite figured out why. But once you > click on that link "Open Metrics URL" and get the "started" splash page, it > all shows it is working. > > I think it has to do with the self-signed certificate and how the browser > wants to be told it is OK. I'm not sure, that's just a guess. > > But, if you started "oc cluster up --metrics" and you see that message, > its all OK. Just click "Open Metrics URL" and see that you can get to the > metrics splash screen via your browser. > > Other than that, I don't have a good explanation - I just know how I get > that message to go away. It shows up for me everytime I "oc cluster up > --metrics". > > ----- Original Message ----- > > Hello > > > > I have tried to run Hawkular in both `oc cluster up` and OpenShift, > different > > versions of both. > > > > The documentation for `oc cluster up` says ( > > https://github.com/openshift/origin/blob/master/docs/ > cluster_up_down.md#installing-metrics > > ) that by using the `--metrics` parameter, metrics is installed. > > > > Its a similar story when installing OpenShift using Openshift-ansible. > > > > However, in both cases, when logging into the OpenShift webconsole I see > an > > error message saying: > > > > An error occurred getting metrics. Open Metrics URL | Don't Show Me Again > > > > > > It would seem there is more to correctly installing Hawkular metrics than > > just using the parameter. > > > > What else needs to be done? And is it possible get the additional info > added > > to the documentation for openshift-ansible and `oc cluster up`? > > > > Thanks > > > > > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170411/3443d933/attachment.html From mazz at redhat.com Tue Apr 11 17:28:49 2017 From: mazz at redhat.com (John Mazzitelli) Date: Tue, 11 Apr 2017 17:28:49 -0400 (EDT) Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. In-Reply-To: References: <2020957409.25663719.1491936939322.JavaMail.zimbra@redhat.com> Message-ID: <1894091182.25699665.1491946129771.JavaMail.zimbra@redhat.com> (you are going to get this reply about 2 to 3 hours after i send it because the mailing list seems to delay my posts - I'm sending this a few minutes after I saw your post though :) If that message is showing up even after, say, 5 minutes, then it sounds like Origin Metrics is failing to start. If you look at your "oc cluster up --metrics" output, see what kind of messages you get regarding starting the metrics component. If you look in your openshift-infra project, you should see the metrics pod - any errors indicated there? In the events tab or logs tab or something like that? Not sure why metrics wouldn't start, but I do know that I have a old machine where it takes a long time for the metrics stuff to start up fully (it has to not only download the image from docker hub the first time, but starting the metrics components can take a while on my slow machine). ----- Original Message ----- > Hi John > > To clarify, its not a certificate issue. After I accept the certificate > warning, the browser redirects to: > > Application is not available > > The application is currently not serving requests at this endpoint. It may > not have been started or is still starting. > > Possible reasons you are seeing this page: > > The host doesn't exist. Make sure the hostname was typed correctly and that > a route matching this hostname exists. > The host exists, but doesn't have a matching path. Check if the URL path > was typed correctly and that the route was created using the desired path. > Route and path matches, but all pods are down. Make sure that the resources > exposed by this route (pods, services, deployment configs, etc) have at > least one pod running. > > > On 11 April 2017 at 20:55, John Mazzitelli wrote: > > > This happens always - and I never quite figured out why. But once you > > click on that link "Open Metrics URL" and get the "started" splash page, it > > all shows it is working. > > > > I think it has to do with the self-signed certificate and how the browser > > wants to be told it is OK. I'm not sure, that's just a guess. > > > > But, if you started "oc cluster up --metrics" and you see that message, > > its all OK. Just click "Open Metrics URL" and see that you can get to the > > metrics splash screen via your browser. > > > > Other than that, I don't have a good explanation - I just know how I get > > that message to go away. It shows up for me everytime I "oc cluster up > > --metrics". > > > > ----- Original Message ----- > > > Hello > > > > > > I have tried to run Hawkular in both `oc cluster up` and OpenShift, > > different > > > versions of both. > > > > > > The documentation for `oc cluster up` says ( > > > https://github.com/openshift/origin/blob/master/docs/ > > cluster_up_down.md#installing-metrics > > > ) that by using the `--metrics` parameter, metrics is installed. > > > > > > Its a similar story when installing OpenShift using Openshift-ansible. > > > > > > However, in both cases, when logging into the OpenShift webconsole I see > > an > > > error message saying: > > > > > > An error occurred getting metrics. Open Metrics URL | Don't Show Me Again > > > > > > > > > It would seem there is more to correctly installing Hawkular metrics than > > > just using the parameter. > > > > > > What else needs to be done? And is it possible get the additional info > > added > > > to the documentation for openshift-ansible and `oc cluster up`? > > > > > > Thanks > > > > > > > > > _______________________________________________ > > > hawkular-dev mailing list > > > hawkular-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > From kurrent93 at gmail.com Wed Apr 12 06:06:50 2017 From: kurrent93 at gmail.com (Anton) Date: Wed, 12 Apr 2017 12:06:50 +0200 Subject: [Hawkular-dev] OpenShift - An error occurred getting metrics. In-Reply-To: <1894091182.25699665.1491946129771.JavaMail.zimbra@redhat.com> References: <2020957409.25663719.1491936939322.JavaMail.zimbra@redhat.com> <1894091182.25699665.1491946129771.JavaMail.zimbra@redhat.com> Message-ID: I thought that the problem might be wtih how my domain name is configured, so I set up a CNAME record to point to the ip of my cluster, but this still did not work. It seems something internally is not working internally with regards to the routing/dns to the hawkular-metrics service. To eleborate, I created a new route, to hawkular-metrics, and this new route DOES work. So I deleted the originally created route, and made a new one, with the same route name - 'hawkular-metrics' - thinking that any openshift deployed services would look up the route name. But it seems this is not the case. It seems it is the route url that is being used. Is it possible to change the url for hawkular-metrics? On 11 April 2017 at 23:28, John Mazzitelli wrote: > (you are going to get this reply about 2 to 3 hours after i send it > because the mailing list seems to delay my posts - I'm sending this a few > minutes after I saw your post though :) > > If that message is showing up even after, say, 5 minutes, then it sounds > like Origin Metrics is failing to start. > > If you look at your "oc cluster up --metrics" output, see what kind of > messages you get regarding starting the metrics component. > > If you look in your openshift-infra project, you should see the metrics > pod - any errors indicated there? In the events tab or logs tab or > something like that? > > Not sure why metrics wouldn't start, but I do know that I have a old > machine where it takes a long time for the metrics stuff to start up fully > (it has to not only download the image from docker hub the first time, but > starting the metrics components can take a while on my slow machine). > > ----- Original Message ----- > > Hi John > > > > To clarify, its not a certificate issue. After I accept the certificate > > warning, the browser redirects to: > > > > Application is not available > > > > The application is currently not serving requests at this endpoint. It > may > > not have been started or is still starting. > > > > Possible reasons you are seeing this page: > > > > The host doesn't exist. Make sure the hostname was typed correctly and > that > > a route matching this hostname exists. > > The host exists, but doesn't have a matching path. Check if the URL path > > was typed correctly and that the route was created using the desired > path. > > Route and path matches, but all pods are down. Make sure that the > resources > > exposed by this route (pods, services, deployment configs, etc) have at > > least one pod running. > > > > > > On 11 April 2017 at 20:55, John Mazzitelli wrote: > > > > > This happens always - and I never quite figured out why. But once you > > > click on that link "Open Metrics URL" and get the "started" splash > page, it > > > all shows it is working. > > > > > > I think it has to do with the self-signed certificate and how the > browser > > > wants to be told it is OK. I'm not sure, that's just a guess. > > > > > > But, if you started "oc cluster up --metrics" and you see that message, > > > its all OK. Just click "Open Metrics URL" and see that you can get to > the > > > metrics splash screen via your browser. > > > > > > Other than that, I don't have a good explanation - I just know how I > get > > > that message to go away. It shows up for me everytime I "oc cluster up > > > --metrics". > > > > > > ----- Original Message ----- > > > > Hello > > > > > > > > I have tried to run Hawkular in both `oc cluster up` and OpenShift, > > > different > > > > versions of both. > > > > > > > > The documentation for `oc cluster up` says ( > > > > https://github.com/openshift/origin/blob/master/docs/ > > > cluster_up_down.md#installing-metrics > > > > ) that by using the `--metrics` parameter, metrics is installed. > > > > > > > > Its a similar story when installing OpenShift using > Openshift-ansible. > > > > > > > > However, in both cases, when logging into the OpenShift webconsole I > see > > > an > > > > error message saying: > > > > > > > > An error occurred getting metrics. Open Metrics URL | Don't Show Me > Again > > > > > > > > > > > > It would seem there is more to correctly installing Hawkular metrics > than > > > > just using the parameter. > > > > > > > > What else needs to be done? And is it possible get the additional > info > > > added > > > > to the documentation for openshift-ansible and `oc cluster up`? > > > > > > > > Thanks > > > > > > > > > > > > _______________________________________________ > > > > hawkular-dev mailing list > > > > hawkular-dev at lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > > > _______________________________________________ > > > hawkular-dev mailing list > > > hawkular-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170412/d4e52185/attachment-0001.html From bheiskell at datto.com Mon Apr 17 18:10:39 2017 From: bheiskell at datto.com (Benjamin Heiskell) Date: Mon, 17 Apr 2017 18:10:39 -0400 Subject: [Hawkular-dev] Proposal for Query DSL Message-ID: Hey all, I recently deployed Hawkular because I found its scaling model appealing. It has been working well for our basic monitoring usage when combined with Grafana. One of my consumers is trying to graph KPIs that involve cross-metric mathematics like sums, division, and moving averages (i.e., post-ingress transformations). I looked into how to accomplish this with Hawkular, but I didn't see any obvious way to do it. The stats endpoint seems to only support aggregations on a single metric/tag at a time. I'm finding that we can't visualize these metrics the same way you can with other TSDBs like Graphite/Prometheus. I did a little research, and found Heiko Rupp's blog post on HawkFX where he seems to already be exploring this idea [1]. Perhaps the query language item on the 2017 road map refers to incorporating this? I would like to assist in developing this feature. Given that there is already a tags DSL, it seems like it would make sense to extend that to support metric transformations. Grafana is also considering adding this to Grafana itself, but it doesn't look like the ticket has any traction [2]. Additionally, shipping potentially megabytes of data to Grafana (or the user's browser) to aggregate doesn't seem ideal. I documented a few TSDB DSLs styles here [3]. What do you all think? Thanks, Ben [1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html [2] https://github.com/grafana/grafana/issues/3677 [3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170417/59dd6129/attachment.html From kurrent93 at gmail.com Mon Apr 17 19:10:47 2017 From: kurrent93 at gmail.com (Anton) Date: Tue, 18 Apr 2017 01:10:47 +0200 Subject: [Hawkular-dev] Proposal for Query DSL In-Reply-To: References: Message-ID: Really great idea! On Apr 18, 2017 1:05 AM, "Benjamin Heiskell" wrote: > Hey all, > > I recently deployed Hawkular because I found its scaling model appealing. > It has been working well for our basic monitoring usage when combined with > Grafana. > > One of my consumers is trying to graph KPIs that involve cross-metric > mathematics like sums, division, and moving averages (i.e., post-ingress > transformations). I looked into how to accomplish this with Hawkular, but I > didn't see any obvious way to do it. The stats endpoint seems to only > support aggregations on a single metric/tag at a time. I'm finding that we > can't visualize these metrics the same way you can with other TSDBs like > Graphite/Prometheus. > > I did a little research, and found Heiko Rupp's blog post on HawkFX where > he seems to already be exploring this idea [1]. Perhaps the query language > item on the 2017 road map refers to incorporating this? > > I would like to assist in developing this feature. Given that there is > already a tags DSL, it seems like it would make sense to extend that to > support metric transformations. > > Grafana is also considering adding this to Grafana itself, but it doesn't > look like the ticket has any traction [2]. Additionally, shipping > potentially megabytes of data to Grafana (or the user's browser) to > aggregate doesn't seem ideal. > > I documented a few TSDB DSLs styles here [3]. > > What do you all think? > > Thanks, > Ben > > [1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html > [2] https://github.com/grafana/grafana/issues/3677 > [3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3 > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170418/7a9d8a40/attachment.html From jshaughn at redhat.com Mon Apr 17 20:01:35 2017 From: jshaughn at redhat.com (Jay Shaughnessy) Date: Mon, 17 Apr 2017 20:01:35 -0400 Subject: [Hawkular-dev] Proposal for Query DSL In-Reply-To: References: Message-ID: <17ecd6a8-068d-df0f-2e62-70bb842f0a3d@redhat.com> On 4/17/2017 6:10 PM, Benjamin Heiskell wrote: > Hey all, > > I recently deployed Hawkular because I found its scaling model > appealing. It has been working well for our basic monitoring usage > when combined with Grafana. > > One of my consumers is trying to graph KPIs that involve cross-metric > mathematics like sums, division, and moving averages (i.e., > post-ingress transformations). I looked into how to accomplish this > with Hawkular, but I didn't see any obvious way to do it. The stats > endpoint seems to only support aggregations on a single metric/tag at > a time. I'm finding that we can't visualize these metrics the same way > you can with other TSDBs like Graphite/Prometheus. It's maybe not what you're looking for but the model for aggregating across metrics is to use a tag query to define the set of applicable metrics. The tag query expressions are fairly robust [1], of course the tags need to be there to make it useful. But given the tags it is possible to do sums, percentiles, etc... As an aside, there is small expression language for using those aggregates for alerting. If you're interested check out http://www.hawkular.org/hawkular-metrics/docs/user-guide/#_query_based_alerting. [1] The tag query language has just recently been released and I'm not sure if the documentation has been updated (Snegrea?). But you can look here https://github.com/hawkular/hawkular-metrics/pull/725. > > I did a little research, and found Heiko Rupp's blog post on HawkFX > where he seems to already be exploring this idea [1]. Perhaps the > query language item on the 2017 road map refers to incorporating this? > > I would like to assist in developing this feature. Given that there is > already a tags DSL, it seems like it would make sense to extend that > to support metric transformations. > > Grafana is also considering adding this to Grafana itself, but it > doesn't look like the ticket has any traction [2]. Additionally, > shipping potentially megabytes of data to Grafana (or the user's > browser) to aggregate doesn't seem ideal. > > I documented a few TSDB DSLs styles here [3]. > > What do you all think? > > Thanks, > Ben > > [1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html > [2] https://github.com/grafana/grafana/issues/3677 > [3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3 > > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170417/e25a57d1/attachment.html From jsanda at redhat.com Mon Apr 17 22:33:50 2017 From: jsanda at redhat.com (John Sanda) Date: Mon, 17 Apr 2017 22:33:50 -0400 Subject: [Hawkular-dev] Proposal for Query DSL In-Reply-To: References: Message-ID: Hi Ben, Comments inline... > On Apr 17, 2017, at 6:10 PM, Benjamin Heiskell wrote: > > Hey all, > > I recently deployed Hawkular because I found its scaling model appealing. It has been working well for our basic monitoring usage when combined with Grafana. > > One of my consumers is trying to graph KPIs that involve cross-metric mathematics like sums, division, and moving averages (i.e., post-ingress transformations). I looked into how to accomplish this with Hawkular, but I didn't see any obvious way to do it. The stats endpoint seems to only support aggregations on a single metric/tag at a time. I'm finding that we can't visualize these metrics the same way you can with other TSDBs like Graphite/Prometheus. The stats endpoints actually do support querying across multiple metrics; however, docs are really lacking which makes it that much easier to miss. I have some docs updates underway. I will try to get to include sections on stats queries. Let me provide a few examples for now. GET /hawkular/metrics/gauges/stats?tags=$tagsQuery where tagsQuery might be like: 'hostname=host1' 'hostname=host1 AND env=stage? 'hostname=h ost1 OR hostname=host2? 'hostname IN [host1, host2, host3]' 'hostname NOT IN [host4, host5]? The /gauges/stats endpoints supports all the other parameter that are supported by /gauges/{id}/stats. There is a corresponding endpoint for counters: GET /hawkular/metrics/counters/stats?tags=$tagsQuery You can also query for rates: GET /hawkular/metrics/counters/rates/stats=$tagsQuery Again, these endpoints all support the same query parameters. There are POST version as well where the query parameters are instead specified as JSON in the request body. POST /hawkular/metrics/gauges/stats/query POST /hawkular/metrics/counters/stats/query The POST endpoints were added in support of the Grafana data source plugin. See https://issues.jboss.org/browse/HWKMETRICS-410 and https://issues.jboss.org/browse/HWKMETRICS-465 for background on the POST endpoints. There are some other endpoints that provide batch query support. I will save the details on those since are not inquiring about them. > > I did a little research, and found Heiko Rupp's blog post on HawkFX where he seems to already be exploring this idea [1]. Perhaps the query language item on the 2017 road map refers to incorporating this? > > I would like to assist in developing this feature. Given that there is already a tags DSL, it seems like it would make sense to extend that to support metric transformations. > > Grafana is also considering adding this to Grafana itself, but it doesn't look like the ticket has any traction [2]. Additionally, shipping potentially megabytes of data to Grafana (or the user's browser) to aggregate doesn't seem ideal. > > I documented a few TSDB DSLs styles here [3]. > > What do you all think? Absolutely agree that Hawkular Metrics would benefit greatly from query language. It would compliment the Grafana plugin nicely. Contributions of any kind, mailing list, IRC (#hawkular) discussion, pull requests, etc, are always welcomed and appreciated. I suggest creating a ticket at https://issues.jboss.org/secure/CreateIssue!default.jspa (be sure to select Hawkular Metrics for the project. > > Thanks, > Ben > > [1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html > [2] https://github.com/grafana/grafana/issues/3677 > [3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3 > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170417/3b7031b3/attachment-0001.html From hrupp at redhat.com Tue Apr 18 04:14:21 2017 From: hrupp at redhat.com (Heiko W.Rupp) Date: Tue, 18 Apr 2017 10:14:21 +0200 Subject: [Hawkular-dev] Proposal for Query DSL In-Reply-To: References: Message-ID: <494DDE53-1EA7-46F0-964B-D93C4FF5396C@redhat.com> On 18 Apr 2017, at 0:10, Benjamin Heiskell wrote: > cross-metric > mathematics like sums, division, and moving averages (i.e., As I wrote in my post, the the blog post in [1] is a great motivator for such a language. > What do you all think? I like that :-) And as others have said already, it is not enough to just use the aggregations over tags, but it should allow for more complete mathematical operations. [1] https://www.circonus.com/2016/06/percentages-arent-people/ From lponce at redhat.com Tue Apr 18 04:25:29 2017 From: lponce at redhat.com (Lucas Ponce) Date: Tue, 18 Apr 2017 10:25:29 +0200 Subject: [Hawkular-dev] Proposal for Query DSL In-Reply-To: <494DDE53-1EA7-46F0-964B-D93C4FF5396C@redhat.com> References: <494DDE53-1EA7-46F0-964B-D93C4FF5396C@redhat.com> Message-ID: On Tue, Apr 18, 2017 at 10:14 AM, Heiko W.Rupp wrote: > On 18 Apr 2017, at 0:10, Benjamin Heiskell wrote: > > > cross-metric > > mathematics like sums, division, and moving averages (i.e., > > As I wrote in my post, the the blog post in [1] is a great motivator > for such a language. > > > What do you all think? > > I like that :-) > > And as others have said already, it is not enough to just use > the aggregations over tags, but it should allow for more complete > mathematical operations. > > > [1] https://www.circonus.com/2016/06/percentages-arent-people/ This post is great. I think that it is good to mention that Hawkular offers support for Nelson Rules [2][3] which provide additional techniques of process control to detect scenarios as described in the post. Modeling is a very interesting and open topic, so offering a rich set of tooling will help users to model better their business scenarios. [2] https://en.wikipedia.org/wiki/Nelson_rules [3] http://www.hawkular.org/docs/rest/rest-alerts.html#NelsonCondition -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170418/03f9b40e/attachment.html From mazz at redhat.com Tue Apr 18 21:53:24 2017 From: mazz at redhat.com (John Mazzitelli) Date: Tue, 18 Apr 2017 21:53:24 -0400 (EDT) Subject: [Hawkular-dev] classifier needed on javaagent as mvn dependency In-Reply-To: <1641418685.28918681.1492566665411.JavaMail.zimbra@redhat.com> Message-ID: <1929211530.28918795.1492566804218.JavaMail.zimbra@redhat.com> If you are pulling in the hawkular javaagent uber-jar as a dependency, you need to use the classifier "shaded" now. We are going to merge the following in the hawkular-agent repo: https://github.com/hawkular/hawkular-agent/pull/318 This will mean the java-agent uber jar will have some classes relocated with the shade plugin. What that further means is that any other maven module pulling in that uber-jar needs to use the new classifier to pick up the jar we expect people to use. See this comment: https://github.com/hawkular/hawkular-agent/pull/318#issuecomment-294837359 org.hawkular.agent hawkular-javaagent ${version.org.hawkular.agent} shaded From miburman at redhat.com Wed Apr 19 08:55:48 2017 From: miburman at redhat.com (Michael Burman) Date: Wed, 19 Apr 2017 15:55:48 +0300 Subject: [Hawkular-dev] Proposal for Query DSL In-Reply-To: References: Message-ID: <62343452-fccd-b87b-eac5-4e94b49e6eae@redhat.com> Hi, If we take the same approach as with tags query language, then some sort of SQL is probably what we'll want - combined with some window functions for time series handling (introduced in SQL:2003). We could perhaps take Apache Calcite as a SQL query engine and implement the necessary functions there - as an example or as a prototype (I don't know the resource requirements of Calcite, but at least we would "easily" get a proper SQL functionality). That same approach is used by Spark SQL to implement SQL query language there (and few other products). The other approach is to also look at some time series query languages that the research has invented who knows how many (none of them have gathered attention - but that doesn't necessarily mean they're bad, just that they were never part of any product). - Micke On 04/18/2017 01:10 AM, Benjamin Heiskell wrote: > Hey all, > > I recently deployed Hawkular because I found its scaling model > appealing. It has been working well for our basic monitoring usage > when combined with Grafana. > > One of my consumers is trying to graph KPIs that involve cross-metric > mathematics like sums, division, and moving averages (i.e., > post-ingress transformations). I looked into how to accomplish this > with Hawkular, but I didn't see any obvious way to do it. The stats > endpoint seems to only support aggregations on a single metric/tag at > a time. I'm finding that we can't visualize these metrics the same way > you can with other TSDBs like Graphite/Prometheus. > > I did a little research, and found Heiko Rupp's blog post on HawkFX > where he seems to already be exploring this idea [1]. Perhaps the > query language item on the 2017 road map refers to incorporating this? > > I would like to assist in developing this feature. Given that there is > already a tags DSL, it seems like it would make sense to extend that > to support metric transformations. > > Grafana is also considering adding this to Grafana itself, but it > doesn't look like the ticket has any traction [2]. Additionally, > shipping potentially megabytes of data to Grafana (or the user's > browser) to aggregate doesn't seem ideal. > > I documented a few TSDB DSLs styles here [3]. > > What do you all think? > > Thanks, > Ben > > [1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html > [2] https://github.com/grafana/grafana/issues/3677 > [3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3 > > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev From mazz at redhat.com Mon Apr 24 21:19:07 2017 From: mazz at redhat.com (John Mazzitelli) Date: Mon, 24 Apr 2017 21:19:07 -0400 (EDT) Subject: [Hawkular-dev] Hawkular Agent 1.0.0.CR1 has been released - inventory in metrics In-Reply-To: <903164902.1388272.1493082713849.JavaMail.zimbra@redhat.com> Message-ID: <1720659720.1389948.1493083147380.JavaMail.zimbra@redhat.com> Hawkular Agent 1.0.0.CR1 has been released. This includes the new "inventory in metrics" feature - Hawkular-Inventory is no longer used to store inventory, the inventory is now stored in Hawkular-Metrics within Cassandra. This is a *significant* change and needs people to beat on it heavily before we can claim victory (hence why it has the CR1 designation and not Final). So, please grab it and use it when you need to use an agent. If you find any bugs, please submit HWKAGENT JIRAs at https://issues.jboss.org/projects/HWKAGENT --John Mazz [this message was sent on April 24, 2017 at 9:19pm EDT] From jsanda at redhat.com Mon Apr 24 23:44:45 2017 From: jsanda at redhat.com (John Sanda) Date: Mon, 24 Apr 2017 23:44:45 -0400 Subject: [Hawkular-dev] Hawkular Agent 1.0.0.CR1 has been released - inventory in metrics In-Reply-To: <1720659720.1389948.1493083147380.JavaMail.zimbra@redhat.com> References: <1720659720.1389948.1493083147380.JavaMail.zimbra@redhat.com> Message-ID: <69002702-BBE3-4F33-B4EB-822B783083DC@redhat.com> A big thanks to Joel and everyone else who helped with the inventory work. By the way, why I am receiving this email on May 1 :) > On Apr 24, 2017, at 9:19 PM, John Mazzitelli wrote: > > Hawkular Agent 1.0.0.CR1 has been released. > > This includes the new "inventory in metrics" feature - Hawkular-Inventory is no longer used to store inventory, the inventory is now stored in Hawkular-Metrics within Cassandra. > > This is a *significant* change and needs people to beat on it heavily before we can claim victory (hence why it has the CR1 designation and not Final). > > So, please grab it and use it when you need to use an agent. > > If you find any bugs, please submit HWKAGENT JIRAs at https://issues.jboss.org/projects/HWKAGENT > > --John Mazz > > [this message was sent on April 24, 2017 at 9:19pm EDT] > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev From mazz at redhat.com Tue Apr 25 12:16:13 2017 From: mazz at redhat.com (John Mazzitelli) Date: Tue, 25 Apr 2017 12:16:13 -0400 (EDT) Subject: [Hawkular-dev] playing with HOSA outside OS In-Reply-To: <1671611459.2099286.1493136378486.JavaMail.zimbra@redhat.com> References: <1671611459.2099286.1493136378486.JavaMail.zimbra@redhat.com> Message-ID: <347317431.2102869.1493136973290.JavaMail.zimbra@redhat.com> I had a couple peeps ask me if they can run HOSA without needing to run it inside an OpenShift cluster (presumably to collect metrics from Prometheus and Jolokia-JMX endpoints that are also running outside of OpenShift). The answer is "yes" and if you are interested, here is a quick how-to. First get a config.yaml used to configure HOSA (that's the wget command below - it just grabs the example config from github) and then run "docker run" to launch HOSA: $ wget -O /tmp/config.yaml https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/master/config.yaml $ docker run --net=host -v /tmp/config.yaml:/config.yaml hawkular/hawkular-openshift-agent --config=/config.yaml This assumes you have Hawkular-Metrics server (or a full Hawkular-Services server) running on 127.0.0.1 listening to port 8080. If not, just edit config.yaml to point to your server. You can edit that config.yaml however you want. By default, HOSA itself is a Prometheus endpoint and will collect its own metrics and store them (see config.yaml for its endpoints definitions). So by running HOSA you will automatically start getting "prometheus" data stored into your H-Metrics. You can add more endpoint definitions to the config to tell HOSA to collect from your own Prometheus and Jolokia-JMX endpoints. You don't have to use docker - if you build the go executable locally (git clone the HOSA repo and "make build") you can run the executable directly. But it's easier to just docker run - no need to git clone, no need to install Go, no need to build anything. From muradkuka at gmail.com Mon Apr 17 08:17:50 2017 From: muradkuka at gmail.com (Mohammad Murad) Date: Mon, 17 Apr 2017 17:47:50 +0530 Subject: [Hawkular-dev] Add functionality to change password. Message-ID: Hello I'm a contributor to the Android Client of Hawkular. On the Gitter channel I suggested that we should give the user an option to change the password. Currently there is no REST API for that. This will be very helpful if the credentials of the user are compromised. Heiko W. Rupp asked me suggest this here. If I can help with this, please let me know. Regards M. Murad GitHub -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170417/cd03931c/attachment-0001.html From hrupp at redhat.com Wed Apr 26 04:09:09 2017 From: hrupp at redhat.com (Heiko W.Rupp) Date: Wed, 26 Apr 2017 10:09:09 +0200 Subject: [Hawkular-dev] Hawkular Agent 1.0.0.CR1 has been released - inventory in metrics In-Reply-To: <1720659720.1389948.1493083147380.JavaMail.zimbra@redhat.com> References: <1720659720.1389948.1493083147380.JavaMail.zimbra@redhat.com> Message-ID: <90854B22-8D6F-43D9-893D-4E1F782A195B@redhat.com> On 25 Apr 2017, at 3:19, John Mazzitelli wrote: > Hawkular Agent 1.0.0.CR1 has been released. > > This includes the new "inventory in metrics" feature - Thanks everyone. But the journey is not yet over .. :) Heiko From hrupp at redhat.com Wed Apr 26 04:33:42 2017 From: hrupp at redhat.com (Heiko W.Rupp) Date: Wed, 26 Apr 2017 10:33:42 +0200 Subject: [Hawkular-dev] Add functionality to change password. In-Reply-To: References: Message-ID: <87F4301A-AA0E-4460-A76C-A1C16C5BC04B@redhat.com> Hey, On 17 Apr 2017, at 14:17, Mohammad Murad wrote: > On the Gitter channel I suggested that we should give the user an option to > change the password. Currently there is no REST API for that. This will be > very helpful if the credentials of the user are compromised. Thank you for bringing this up. I agree that there should be some way of doing that, but as it stands right now it is also tricky from a technical standpoint. One part is on the Hawkular-services server itself to change the password. The other part is also to convey this change to connected agents. And in the case that they are running in (Docker) Containers also to propagate the change there (be it for a new image build, be it to set the environment accordingly). On the server we currently set up credentials via add-user on startup (if I recall correctly). This would also need to change. If the agent is mutable, we could probably forward the password change via the operations (web-socket) to the agents, which store the new credentials and use them from there on. The immutable agents (in Containers) could use the same mechanisms to change the in-memory credentials for the running agent. Leaves us with procedures to change the environment/image Heiko From mazz at redhat.com Wed Apr 26 12:10:14 2017 From: mazz at redhat.com (John Mazzitelli) Date: Wed, 26 Apr 2017 12:10:14 -0400 (EDT) Subject: [Hawkular-dev] need a briefing on what the hawkular agent is now doing with respect to inventory In-Reply-To: <874879402.3039474.1493222969465.JavaMail.zimbra@redhat.com> Message-ID: <239530940.3039636.1493223014944.JavaMail.zimbra@redhat.com> Joel, Now that the new inventory-into-metrics is in master and released, I need to know what you did :-D I suspect others will want to know what you did too. Is it possible for you to write something up or have a 15-minute Blue Jeans session to discuss how inventory is stored in H-Metrics? I am going to need to know this because I have to implement it in GoLang for HOSA, unless you want to do it :) --John Mazz From mazz at redhat.com Wed Apr 26 12:35:08 2017 From: mazz at redhat.com (John Mazzitelli) Date: Wed, 26 Apr 2017 12:35:08 -0400 (EDT) Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: <1345476794.3061617.1493223725176.JavaMail.zimbra@redhat.com> Message-ID: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> The first use-case for needing HOSA to collect and store inventory has come in - from the Fuse team, specifically. This email is going to explain the current plan to get this to happen. Feel free to chime in with thoughts. Right now, HOSA does not support collecting and storing inventory. HJA (and HWFA) supports collecting and storing inventory along with metrics. HOSA has the privileges necessary to write directly to Origin Metrics. HJA does not. We need to combine these to get what we need (and we need this soon because Fuse needs it). Currently, the idea is: a) HJA will have a new storage adapter mode "HOSA" (right now it has "HAWKULAR" and "METRICS" - the former is to support running with a full hawkular server such as when running within CloudForms and the latter is for when running with just a H-Metrics server). With this new HOSA mode, HJA will not store metrics or inventory directly into H-Metrics. Instead it will "cache" the data and wait for HOSA to come and ask for the data. b) HOSA will have a new endpoint type "hawkular" (right now it has "prometheus" and "jolokia" and "json"). This new type will tell HOSA to read data from some HJA (which will including inventory data) and HOSA will simply pass that data on to Origin Metrics. HOSA will read both metrics and inventory from HJA and HOSA will store that data directly to Origin Metrics essentially making HOSA a proxy to Origin Metrics. I don't know if HOSA will decorate the data with its own metadata (like tags and things) or if it will truly be a pass-through. I suspect HOSA will need to massage the data it gets from HJA to ensure the data IDs are unique across the OpenShift cluster and things like that. That's the overall idea. I'm sure there are going to be stumbling blocks along the way that is going to force us to change some things around. But at a high level, I think it should work. HJA caches inventory and metrics rather than writing directly to an H-Metrics server - and HOSA will collect that cached data in HJA and it will be the one to store it in Origin Metrics. From miburman at redhat.com Wed Apr 26 14:41:59 2017 From: miburman at redhat.com (Michael Burman) Date: Wed, 26 Apr 2017 21:41:59 +0300 Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> References: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> Message-ID: <446794ea-cd49-b544-3724-acd10ef47d68@redhat.com> Why implement it in HOSA instead of creating an inventory module to the Golang client? That way some other applications could use the inventory featureset also in the future (without needing to go through HOSA). HOSA could then of course consume the client, like it does with the metrics client. - Micke On 04/26/2017 07:35 PM, John Mazzitelli wrote: > The first use-case for needing HOSA to collect and store inventory has come in - from the Fuse team, specifically. This email is going to explain the current plan to get this to happen. Feel free to chime in with thoughts. > > Right now, HOSA does not support collecting and storing inventory. HJA (and HWFA) supports collecting and storing inventory along with metrics. > > HOSA has the privileges necessary to write directly to Origin Metrics. HJA does not. > > We need to combine these to get what we need (and we need this soon because Fuse needs it). > > Currently, the idea is: > > a) HJA will have a new storage adapter mode "HOSA" (right now it has "HAWKULAR" and "METRICS" - the former is to support running with a full hawkular server such as when running within CloudForms and the latter is for when running with just a H-Metrics server). With this new HOSA mode, HJA will not store metrics or inventory directly into H-Metrics. Instead it will "cache" the data and wait for HOSA to come and ask for the data. > > b) HOSA will have a new endpoint type "hawkular" (right now it has "prometheus" and "jolokia" and "json"). This new type will tell HOSA to read data from some HJA (which will including inventory data) and HOSA will simply pass that data on to Origin Metrics. HOSA will read both metrics and inventory from HJA and HOSA will store that data directly to Origin Metrics essentially making HOSA a proxy to Origin Metrics. > > I don't know if HOSA will decorate the data with its own metadata (like tags and things) or if it will truly be a pass-through. I suspect HOSA will need to massage the data it gets from HJA to ensure the data IDs are unique across the OpenShift cluster and things like that. > > That's the overall idea. I'm sure there are going to be stumbling blocks along the way that is going to force us to change some things around. But at a high level, I think it should work. HJA caches inventory and metrics rather than writing directly to an H-Metrics server - and HOSA will collect that cached data in HJA and it will be the one to store it in Origin Metrics. > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev From miburman at redhat.com Wed Apr 26 14:44:20 2017 From: miburman at redhat.com (Michael Burman) Date: Wed, 26 Apr 2017 21:44:20 +0300 Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> References: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> Message-ID: <384747a5-79f1-fa4d-8bee-fb191212dc44@redhat.com> I would have replied to my own mail, but I have no idea where it's actually surfing at the moment. Perhaps it's using NCSA Mosaic to load Gmail. In any case, second thing, why is only HOSA allowed to contact the metrics and not the Java-agent? Perhaps we should also consider allowing it to report directly to the metrics instance (both ways could be supported). - Micke On 04/26/2017 07:35 PM, John Mazzitelli wrote: > The first use-case for needing HOSA to collect and store inventory has come in - from the Fuse team, specifically. This email is going to explain the current plan to get this to happen. Feel free to chime in with thoughts. > > Right now, HOSA does not support collecting and storing inventory. HJA (and HWFA) supports collecting and storing inventory along with metrics. > > HOSA has the privileges necessary to write directly to Origin Metrics. HJA does not. > > We need to combine these to get what we need (and we need this soon because Fuse needs it). > > Currently, the idea is: > > a) HJA will have a new storage adapter mode "HOSA" (right now it has "HAWKULAR" and "METRICS" - the former is to support running with a full hawkular server such as when running within CloudForms and the latter is for when running with just a H-Metrics server). With this new HOSA mode, HJA will not store metrics or inventory directly into H-Metrics. Instead it will "cache" the data and wait for HOSA to come and ask for the data. > > b) HOSA will have a new endpoint type "hawkular" (right now it has "prometheus" and "jolokia" and "json"). This new type will tell HOSA to read data from some HJA (which will including inventory data) and HOSA will simply pass that data on to Origin Metrics. HOSA will read both metrics and inventory from HJA and HOSA will store that data directly to Origin Metrics essentially making HOSA a proxy to Origin Metrics. > > I don't know if HOSA will decorate the data with its own metadata (like tags and things) or if it will truly be a pass-through. I suspect HOSA will need to massage the data it gets from HJA to ensure the data IDs are unique across the OpenShift cluster and things like that. > > That's the overall idea. I'm sure there are going to be stumbling blocks along the way that is going to force us to change some things around. But at a high level, I think it should work. HJA caches inventory and metrics rather than writing directly to an H-Metrics server - and HOSA will collect that cached data in HJA and it will be the one to store it in Origin Metrics. > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev From mazz at redhat.com Wed Apr 26 16:10:48 2017 From: mazz at redhat.com (John Mazzitelli) Date: Wed, 26 Apr 2017 16:10:48 -0400 (EDT) Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: <446794ea-cd49-b544-3724-acd10ef47d68@redhat.com> References: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> <446794ea-cd49-b544-3724-acd10ef47d68@redhat.com> Message-ID: <565429390.3165064.1493237448795.JavaMail.zimbra@redhat.com> Sounds good to me! Who's in charge of the golang client? ;) ----- Original Message ----- > Why implement it in HOSA instead of creating an inventory module to the > Golang client? That way some other applications could use the inventory > featureset also in the future (without needing to go through HOSA). > > HOSA could then of course consume the client, like it does with the > metrics client. > > - Micke > > > On 04/26/2017 07:35 PM, John Mazzitelli wrote: > > The first use-case for needing HOSA to collect and store inventory has come > > in - from the Fuse team, specifically. This email is going to explain the > > current plan to get this to happen. Feel free to chime in with thoughts. > > > > Right now, HOSA does not support collecting and storing inventory. HJA (and > > HWFA) supports collecting and storing inventory along with metrics. > > > > HOSA has the privileges necessary to write directly to Origin Metrics. HJA > > does not. > > > > We need to combine these to get what we need (and we need this soon because > > Fuse needs it). > > > > Currently, the idea is: > > > > a) HJA will have a new storage adapter mode "HOSA" (right now it has > > "HAWKULAR" and "METRICS" - the former is to support running with a full > > hawkular server such as when running within CloudForms and the latter is > > for when running with just a H-Metrics server). With this new HOSA mode, > > HJA will not store metrics or inventory directly into H-Metrics. Instead > > it will "cache" the data and wait for HOSA to come and ask for the data. > > > > b) HOSA will have a new endpoint type "hawkular" (right now it has > > "prometheus" and "jolokia" and "json"). This new type will tell HOSA to > > read data from some HJA (which will including inventory data) and HOSA > > will simply pass that data on to Origin Metrics. HOSA will read both > > metrics and inventory from HJA and HOSA will store that data directly to > > Origin Metrics essentially making HOSA a proxy to Origin Metrics. > > > > I don't know if HOSA will decorate the data with its own metadata (like > > tags and things) or if it will truly be a pass-through. I suspect HOSA > > will need to massage the data it gets from HJA to ensure the data IDs are > > unique across the OpenShift cluster and things like that. > > > > That's the overall idea. I'm sure there are going to be stumbling blocks > > along the way that is going to force us to change some things around. But > > at a high level, I think it should work. HJA caches inventory and metrics > > rather than writing directly to an H-Metrics server - and HOSA will > > collect that cached data in HJA and it will be the one to store it in > > Origin Metrics. > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > From mazz at redhat.com Wed Apr 26 16:12:30 2017 From: mazz at redhat.com (John Mazzitelli) Date: Wed, 26 Apr 2017 16:12:30 -0400 (EDT) Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: <384747a5-79f1-fa4d-8bee-fb191212dc44@redhat.com> References: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> <384747a5-79f1-fa4d-8bee-fb191212dc44@redhat.com> Message-ID: <476709290.3165329.1493237550085.JavaMail.zimbra@redhat.com> Because Origin Metrics is locked down. Without special privileges (that only HOSA and Heapster have) no one can write to the Origin Metrics that is installed in the OpenShift cluster. And, yes, we asked. :) This was plan B. ----- Original Message ----- > I would have replied to my own mail, but I have no idea where it's > actually surfing at the moment. Perhaps it's using NCSA Mosaic to load > Gmail. > > In any case, second thing, why is only HOSA allowed to contact the > metrics and not the Java-agent? Perhaps we should also consider allowing > it to report directly to the metrics instance (both ways could be > supported). > > - Micke > > > On 04/26/2017 07:35 PM, John Mazzitelli wrote: > > The first use-case for needing HOSA to collect and store inventory has come > > in - from the Fuse team, specifically. This email is going to explain the > > current plan to get this to happen. Feel free to chime in with thoughts. > > > > Right now, HOSA does not support collecting and storing inventory. HJA (and > > HWFA) supports collecting and storing inventory along with metrics. > > > > HOSA has the privileges necessary to write directly to Origin Metrics. HJA > > does not. > > > > We need to combine these to get what we need (and we need this soon because > > Fuse needs it). > > > > Currently, the idea is: > > > > a) HJA will have a new storage adapter mode "HOSA" (right now it has > > "HAWKULAR" and "METRICS" - the former is to support running with a full > > hawkular server such as when running within CloudForms and the latter is > > for when running with just a H-Metrics server). With this new HOSA mode, > > HJA will not store metrics or inventory directly into H-Metrics. Instead > > it will "cache" the data and wait for HOSA to come and ask for the data. > > > > b) HOSA will have a new endpoint type "hawkular" (right now it has > > "prometheus" and "jolokia" and "json"). This new type will tell HOSA to > > read data from some HJA (which will including inventory data) and HOSA > > will simply pass that data on to Origin Metrics. HOSA will read both > > metrics and inventory from HJA and HOSA will store that data directly to > > Origin Metrics essentially making HOSA a proxy to Origin Metrics. > > > > I don't know if HOSA will decorate the data with its own metadata (like > > tags and things) or if it will truly be a pass-through. I suspect HOSA > > will need to massage the data it gets from HJA to ensure the data IDs are > > unique across the OpenShift cluster and things like that. > > > > That's the overall idea. I'm sure there are going to be stumbling blocks > > along the way that is going to force us to change some things around. But > > at a high level, I think it should work. HJA caches inventory and metrics > > rather than writing directly to an H-Metrics server - and HOSA will > > collect that cached data in HJA and it will be the one to store it in > > Origin Metrics. > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > From hrupp at redhat.com Thu Apr 27 04:04:36 2017 From: hrupp at redhat.com (Heiko W.Rupp) Date: Thu, 27 Apr 2017 10:04:36 +0200 Subject: [Hawkular-dev] HEADS UP: Hawkular Services is now on Inventory.v3 Message-ID: <8B112884-405D-433D-83F8-F111E1955C5E@redhat.com> Hey, sorry for not sending this email earlier. Hawkular-services has in master and also release 0.36 switched to new Inventory.v3, which is Inventory on Metrics. The old Inventory component (that uses a relational DB) has been removed. The Ruby client gem has been bumped to version 3.0.1 to cater for this change [1]. Agent(s) 1.0.cr1 are also using this new Inventory code. Joel is currently preparing a show&tell to walk us through the new Inventory version and will also update docs on hawkular.org accordingly. My thanks goes to everyone involved to make this happen and especially to Joel who did the majority of the work. Heiko [1] And HawkFX in master is also using that gem version now. -- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht M?nchen HRB 153243 Gesch?ftsf?hrer: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander From jtakvori at redhat.com Thu Apr 27 05:53:27 2017 From: jtakvori at redhat.com (Joel Takvorian) Date: Thu, 27 Apr 2017 11:53:27 +0200 Subject: [Hawkular-dev] need a briefing on what the hawkular agent is now doing with respect to inventory In-Reply-To: <239530940.3039636.1493223014944.JavaMail.zimbra@redhat.com> References: <874879402.3039474.1493222969465.JavaMail.zimbra@redhat.com> <239530940.3039636.1493223014944.JavaMail.zimbra@redhat.com> Message-ID: Sure, I'm currently preparing a documentation and a slide show. I'd be glad to do some golang for the first time, by the way :) On Wed, Apr 26, 2017 at 6:10 PM, John Mazzitelli wrote: > Joel, > > Now that the new inventory-into-metrics is in master and released, I need > to know what you did :-D I suspect others will want to know what you did > too. > > Is it possible for you to write something up or have a 15-minute Blue > Jeans session to discuss how inventory is stored in H-Metrics? > > I am going to need to know this because I have to implement it in GoLang > for HOSA, unless you want to do it :) > > --John Mazz > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170427/9b4917dc/attachment.html From lmi at mailme.dk Thu Apr 27 10:45:15 2017 From: lmi at mailme.dk (Lars Milland) Date: Thu, 27 Apr 2017 16:45:15 +0200 Subject: [Hawkular-dev] OpenShift OAuth authentication and authorization for Hawkular APM Message-ID: <007701d2bf64$e24ba9d0$a6e2fd70$@mailme.dk> Hi It would be really great if a functionality for Hawkular APM could be found/established, matching the one that exists for Hawkular Metrics wise for OpenShift, where the metrics are stored per tenant/namespace, and then Hawkular security wise is integrated to the OAuth based security model of OpenShift. Is that a requirement/feature that have been considered? Or would it maybe already be possible to integrate the Hawkular APM components to OpenShift OAuth based security. Even if the Hawkular APM storage and security model would not fit to the fully multitenant way of OpenShift, if just the security model of a Hawkular APM installation could be connected to the OpenShift OAuth model, then one Hawkular APM instance could be setup with "service account tokens" used for sending metrics to the instance, and users could log into the Hawkular APM UI with again OpenShift OAuth managed credentials, mapped to roles coming from the OAuth ticket. Much the same way that the security model of the OpenShift integrated Jenkins works - see: https://github.com/openshift/jenkins-openshift-login-plugin The current security model of APM is rather limited as far as I understand - and based solely on a single manually fixed username/password for both contributing application performance metrics/log entries, and same for the Hawkular APM UI. Best regards Lars Milland -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170427/3a084e94/attachment.html From jpkroehling at redhat.com Thu Apr 27 11:57:47 2017 From: jpkroehling at redhat.com (=?UTF-8?Q?Juraci_Paix=c3=a3o_Kr=c3=b6hling?=) Date: Thu, 27 Apr 2017 17:57:47 +0200 Subject: [Hawkular-dev] OpenShift OAuth authentication and authorization for Hawkular APM In-Reply-To: <007701d2bf64$e24ba9d0$a6e2fd70$@mailme.dk> References: <007701d2bf64$e24ba9d0$a6e2fd70$@mailme.dk> Message-ID: <1d83c40f-460d-429e-30eb-02e23e88eb17@redhat.com> Lars, Indeed, the security mechanism for APM is very simple at the moment: it uses JAAS, which is concretely implemented by the file-based auth from Wildfly. The idea was that Red Hat SSO / Keycloak could be used in a production setup, as the Wildfly Adapter is also implementing JAAS. In fact, Keycloak *was* used in a previous iteration of the project. At this point, however, we are focused on collaborating with Jaeger: http://www.hawkular.org/blog/2017/04/19/hawkular-apm-jaeger.html Even though we have not started discussing yet how we'll manage security in Jaeger, I chatted with some colleagues from the Keycloak team last week to see whether or not the usage of Keycloak Proxy as a sidecar would make sense in an OpenShift deployment. There's no conclusion yet, but it's something to be tested :) At this moment, we have other priorities for Jaeger and the Keycloak team is also busy with other tasks, but if this is a topic you'd be interested in contributing, I'd be more than happy to share what I have in mind. - Juca. On 04/27/2017 04:45 PM, Lars Milland wrote: > Hi > > It would be really great if a functionality for Hawkular APM could be > found/established, matching the one that exists for Hawkular Metrics > wise for OpenShift, where the metrics are stored per tenant/namespace, > and then Hawkular security wise is integrated to the OAuth based > security model of OpenShift. > > Is that a requirement/feature that have been considered? Or would it > maybe already be possible to integrate the Hawkular APM components to > OpenShift OAuth based security. Even if the Hawkular APM storage and > security model would not fit to the fully multitenant way of OpenShift, > if just the security model of a Hawkular APM installation could be > connected to the OpenShift OAuth model, then one Hawkular APM instance > could be setup with ?service account tokens? used for sending metrics to > the instance, and users could log into the Hawkular APM UI with again > OpenShift OAuth managed credentials, mapped to roles coming from the > OAuth ticket. Much the same way that the security model of the OpenShift > integrated Jenkins works - see: > > https://github.com/openshift/jenkins-openshift-login-plugin > > The current security model of APM is rather limited as far as I > understand ? and based solely on a single manually fixed > username/password for both contributing application performance > metrics/log entries, and same for the Hawkular APM UI. > > Best regards > > Lars Milland > > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > From jshaughn at redhat.com Thu Apr 27 15:31:08 2017 From: jshaughn at redhat.com (Jay Shaughnessy) Date: Thu, 27 Apr 2017 15:31:08 -0400 Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: <476709290.3165329.1493237550085.JavaMail.zimbra@redhat.com> References: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> <384747a5-79f1-fa4d-8bee-fb191212dc44@redhat.com> <476709290.3165329.1493237550085.JavaMail.zimbra@redhat.com> Message-ID: Despite this adding another thing to install and coordinate (HJA), it seems like it could work. The discovery and all the config support is already impl'd in HJA, so no need to redo it in HOSA. And adding new endpoint support shouldn't be too big. HJA in HOSA mode couldn't/shouldn't cache too much, right? The inventory is likely not huge and is already in memory, but metrics that aren't consumed quickly should likely be discarded. I agree with Micke, we (and by we I probably mean Joel) should add required inv support in the GoLang client, similar to what we already have in the ruby client and some Java "helpers". HJA would not have any ability to push to HOSA, right? So changes in inventory would likely have a [short] delay before being pulled by HOSA. On 4/26/2017 4:12 PM, John Mazzitelli wrote: > Because Origin Metrics is locked down. Without special privileges (that only HOSA and Heapster have) no one can write to the Origin Metrics that is installed in the OpenShift cluster. > > And, yes, we asked. :) This was plan B. > > ----- Original Message ----- >> I would have replied to my own mail, but I have no idea where it's >> actually surfing at the moment. Perhaps it's using NCSA Mosaic to load >> Gmail. >> >> In any case, second thing, why is only HOSA allowed to contact the >> metrics and not the Java-agent? Perhaps we should also consider allowing >> it to report directly to the metrics instance (both ways could be >> supported). >> >> - Micke >> >> >> On 04/26/2017 07:35 PM, John Mazzitelli wrote: >>> The first use-case for needing HOSA to collect and store inventory has come >>> in - from the Fuse team, specifically. This email is going to explain the >>> current plan to get this to happen. Feel free to chime in with thoughts. >>> >>> Right now, HOSA does not support collecting and storing inventory. HJA (and >>> HWFA) supports collecting and storing inventory along with metrics. >>> >>> HOSA has the privileges necessary to write directly to Origin Metrics. HJA >>> does not. >>> >>> We need to combine these to get what we need (and we need this soon because >>> Fuse needs it). >>> >>> Currently, the idea is: >>> >>> a) HJA will have a new storage adapter mode "HOSA" (right now it has >>> "HAWKULAR" and "METRICS" - the former is to support running with a full >>> hawkular server such as when running within CloudForms and the latter is >>> for when running with just a H-Metrics server). With this new HOSA mode, >>> HJA will not store metrics or inventory directly into H-Metrics. Instead >>> it will "cache" the data and wait for HOSA to come and ask for the data. >>> >>> b) HOSA will have a new endpoint type "hawkular" (right now it has >>> "prometheus" and "jolokia" and "json"). This new type will tell HOSA to >>> read data from some HJA (which will including inventory data) and HOSA >>> will simply pass that data on to Origin Metrics. HOSA will read both >>> metrics and inventory from HJA and HOSA will store that data directly to >>> Origin Metrics essentially making HOSA a proxy to Origin Metrics. >>> >>> I don't know if HOSA will decorate the data with its own metadata (like >>> tags and things) or if it will truly be a pass-through. I suspect HOSA >>> will need to massage the data it gets from HJA to ensure the data IDs are >>> unique across the OpenShift cluster and things like that. >>> >>> That's the overall idea. I'm sure there are going to be stumbling blocks >>> along the way that is going to force us to change some things around. But >>> at a high level, I think it should work. HJA caches inventory and metrics >>> rather than writing directly to an H-Metrics server - and HOSA will >>> collect that cached data in HJA and it will be the one to store it in >>> Origin Metrics. >>> _______________________________________________ >>> hawkular-dev mailing list >>> hawkular-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/hawkular-dev >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev >> > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170427/85762c8b/attachment-0001.html From mazz at redhat.com Thu Apr 27 15:47:01 2017 From: mazz at redhat.com (John Mazzitelli) Date: Thu, 27 Apr 2017 15:47:01 -0400 (EDT) Subject: [Hawkular-dev] bug in MiQ / javaagent integration - needs to be fixed In-Reply-To: <679409956.3757890.1493322048845.JavaMail.zimbra@redhat.com> Message-ID: <532039521.3759659.1493322421095.JavaMail.zimbra@redhat.com> There is a problem with the new Java agent and MiQ. This problem is because the agent is no longer in WildFly's DMR tree of resources (because only WildFly subsystems are in the DMR tree). Look here: https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L596-L603 So when the agent (java or subsystem agent, doesn't matter) is monitoring EAP over the DMR management interface, that's the metadata it uses. You can see we inject two non-WildFly attributes (immutable and in-container) under the WildFly Server type. But these come from the agent - hence why you see: path: /subsystem=hawkular-wildfly-agent But when running this as a java agent, that resource doesn't exist - there is no agent subsystem anymore. So the immutable and in-container attributes end up not getting defined under the WildFly Server resource. The problem is MiQ looks for those attributes under WildFly Servers to determine if the agent is in immutable mode and if its running in a container. In effect, we are trying to inject under a WildFly Server resource completely unrelated information (immutable and in-container attributes) - those are related to the AGENT, not the WildFly Server itself (i.e. you won't see those two attributes if you use the jboss-cli under the root "/" resource). We probably should have had MiQ examine the AGENT resource for this (since "immutable" and "in container" are attributes on the agent, not the server). And we shouldn't define "immutable" and "in container" under the WildFly Server resource because that is depending on the fact that the agent is running as a subsystem. As a Java Agent, as you see, those attributes will never show any values because there is no agent subsystem. MiQ can look at the agent attributes because the agent itself exposes immutable and in-container (because they ARE actual properties the agent has) in inventory. Under the agent resource in inventory you will see these: https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L983-L989 Note that we DO expose immutable and in-container under the Agent JMX resource in inventory also - and it has the same resource name as the subsystem agent (the name as you will see it in inventory): https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L1391-L1399 Hence why I think the best/easiest thing to do is get MiQ to examine the attributes on the resource where they really belong (and we should remove them from the WildFly Server type). From mazz at redhat.com Thu Apr 27 16:09:53 2017 From: mazz at redhat.com (John Mazzitelli) Date: Thu, 27 Apr 2017 16:09:53 -0400 (EDT) Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: References: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> <384747a5-79f1-fa4d-8bee-fb191212dc44@redhat.com> <476709290.3165329.1493237550085.JavaMail.zimbra@redhat.com> Message-ID: <1623540796.3765173.1493323793895.JavaMail.zimbra@redhat.com> > but metrics that aren't consumed quickly should likely be discarded. Yes, I think we rely on HOSA to collect metrics on its intervals, not HJA. HJA will just maintain the last known metric data point for each metric. > HJA would not have any ability to push to HOSA, right? So changes in inventory would likely have a [short] delay > before being pulled by HOSA. Yes, good point. HJA does not push to HOSA. From jshaughn at redhat.com Thu Apr 27 17:04:50 2017 From: jshaughn at redhat.com (Jay Shaughnessy) Date: Thu, 27 Apr 2017 17:04:50 -0400 Subject: [Hawkular-dev] bug in MiQ / javaagent integration - needs to be fixed In-Reply-To: <532039521.3759659.1493322421095.JavaMail.zimbra@redhat.com> References: <532039521.3759659.1493322421095.JavaMail.zimbra@redhat.com> Message-ID: <92eeafff-af54-5faa-2d8b-05dac69bd334@redhat.com> Is it easy to determine which agent resource maps to the server of interest? Perhaps from the feedId on the server resource? On 4/27/2017 3:47 PM, John Mazzitelli wrote: > There is a problem with the new Java agent and MiQ. > > This problem is because the agent is no longer in WildFly's DMR tree of resources (because only WildFly subsystems are in the DMR tree). > > Look here: > > https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L596-L603 > > So when the agent (java or subsystem agent, doesn't matter) is monitoring EAP over the DMR management interface, that's the metadata it uses. You can see we inject two non-WildFly attributes (immutable and in-container) under the WildFly Server type. But these come from the agent - hence why you see: > > path: /subsystem=hawkular-wildfly-agent > > But when running this as a java agent, that resource doesn't exist - there is no agent subsystem anymore. So the immutable and in-container attributes end up not getting defined under the WildFly Server resource. > > The problem is MiQ looks for those attributes under WildFly Servers to determine if the agent is in immutable mode and if its running in a container. > > In effect, we are trying to inject under a WildFly Server resource completely unrelated information (immutable and in-container attributes) - those are related to the AGENT, not the WildFly Server itself (i.e. you won't see those two attributes if you use the jboss-cli under the root "/" resource). > > We probably should have had MiQ examine the AGENT resource for this (since "immutable" and "in container" are attributes on the agent, not the server). And we shouldn't define "immutable" and "in container" under the WildFly Server resource because that is depending on the fact that the agent is running as a subsystem. As a Java Agent, as you see, those attributes will never show any values because there is no agent subsystem. > > MiQ can look at the agent attributes because the agent itself exposes immutable and in-container (because they ARE actual properties the agent has) in inventory. Under the agent resource in inventory you will see these: > > https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L983-L989 > > Note that we DO expose immutable and in-container under the Agent JMX resource in inventory also - and it has the same resource name as the subsystem agent (the name as you will see it in inventory): > > https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L1391-L1399 > > Hence why I think the best/easiest thing to do is get MiQ to examine the attributes on the resource where they really belong (and we should remove them from the WildFly Server type). > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170427/db0a23fd/attachment.html From mazz at redhat.com Thu Apr 27 20:31:58 2017 From: mazz at redhat.com (John Mazzitelli) Date: Thu, 27 Apr 2017 20:31:58 -0400 (EDT) Subject: [Hawkular-dev] bug in MiQ / javaagent integration - needs to be fixed In-Reply-To: <92eeafff-af54-5faa-2d8b-05dac69bd334@redhat.com> References: <532039521.3759659.1493322421095.JavaMail.zimbra@redhat.com> <92eeafff-af54-5faa-2d8b-05dac69bd334@redhat.com> Message-ID: <742502083.3869461.1493339518821.JavaMail.zimbra@redhat.com> > Is it easy to determine which agent resource maps to the server of interest? > Perhaps from the feedId on the server resource? Yup - you use the feed ID. Feed ID of the WildFly Server is the same feed ID of the agent - and there is only one singleton agent per feed ID. So should be easy to find it. > > On 4/27/2017 3:47 PM, John Mazzitelli wrote: > > There is a problem with the new Java agent and MiQ. > > This problem is because the agent is no longer in WildFly's DMR tree of > resources (because only WildFly subsystems are in the DMR tree). > > Look here: > https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L596-L603 > So when the agent (java or subsystem agent, doesn't matter) is monitoring > EAP over the DMR management interface, that's the metadata it uses. You can > see we inject two non-WildFly attributes (immutable and in-container) under > the WildFly Server type. But these come from the agent - hence why you see: > > path: /subsystem=hawkular-wildfly-agent > > But when running this as a java agent, that resource doesn't exist - there is > no agent subsystem anymore. So the immutable and in-container attributes end > up not getting defined under the WildFly Server resource. > > The problem is MiQ looks for those attributes under WildFly Servers to > determine if the agent is in immutable mode and if its running in a > container. > > In effect, we are trying to inject under a WildFly Server resource completely > unrelated information (immutable and in-container attributes) - those are > related to the AGENT, not the WildFly Server itself (i.e. you won't see > those two attributes if you use the jboss-cli under the root "/" resource). > > We probably should have had MiQ examine the AGENT resource for this (since > "immutable" and "in container" are attributes on the agent, not the server). > And we shouldn't define "immutable" and "in container" under the WildFly > Server resource because that is depending on the fact that the agent is > running as a subsystem. As a Java Agent, as you see, those attributes will > never show any values because there is no agent subsystem. > > MiQ can look at the agent attributes because the agent itself exposes > immutable and in-container (because they ARE actual properties the agent > has) in inventory. Under the agent resource in inventory you will see these: > https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L983-L989 > Note that we DO expose immutable and in-container under the Agent JMX > resource in inventory also - and it has the same resource name as the > subsystem agent (the name as you will see it in inventory): > https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent-wildfly-feature-pack/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml#L1391-L1399 > Hence why I think the best/easiest thing to do is get MiQ to examine the > attributes on the resource where they really belong (and we should remove > them from the WildFly Server type). From hrupp at redhat.com Fri Apr 28 03:45:08 2017 From: hrupp at redhat.com (Heiko W.Rupp) Date: Fri, 28 Apr 2017 09:45:08 +0200 Subject: [Hawkular-dev] HEADS UP: Hawkular Services is now on Inventory.v3 In-Reply-To: <8B112884-405D-433D-83F8-F111E1955C5E@redhat.com> References: <8B112884-405D-433D-83F8-F111E1955C5E@redhat.com> Message-ID: <11F3ED36-6A87-4423-BC89-AF78F770F2A6@redhat.com> Hey, the changes have now also been merged into ManageIQ master, so we are on inventory.v3 end-to-end Joel has started a tech document for the Hawkular.org pages you can find the PR here: https://github.com/hawkular/hawkular.github.io/pull/311 Thanks Heiko On 27 Apr 2017, at 10:04, Heiko W.Rupp wrote: > Hey, > > sorry for not sending this email earlier. > > Hawkular-services has in master and also release 0.36 > switched to new Inventory.v3, which is Inventory on Metrics. > The old Inventory component (that uses a relational DB) > has been removed. > The Ruby client gem has been bumped to version 3.0.1 > to cater for this change [1]. > Agent(s) 1.0.cr1 are also using this new Inventory code. > > Joel is currently preparing a show&tell to walk us through > the new Inventory version and will also update docs on > hawkular.org accordingly. > > My thanks goes to everyone involved to make this happen > and especially to Joel who did the majority of the work. > > Heiko > > [1] And HawkFX in master is also using that gem version now. > -- > Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, > Werner-von-Siemens-Ring 14, D-85630 Grasbrunn > Handelsregister: Amtsgericht M?nchen HRB 153243 > Gesch?ftsf?hrer: Charles Cachera, Michael Cunningham, Michael > O'Neill, > Eric Shander > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev -- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht M?nchen HRB 153243 Gesch?ftsf?hrer: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander From theute at redhat.com Fri Apr 28 05:39:28 2017 From: theute at redhat.com (Thomas Heute) Date: Fri, 28 Apr 2017 11:39:28 +0200 Subject: [Hawkular-dev] HEADS UP: Hawkular Services is now on Inventory.v3 In-Reply-To: <11F3ED36-6A87-4423-BC89-AF78F770F2A6@redhat.com> References: <8B112884-405D-433D-83F8-F111E1955C5E@redhat.com> <11F3ED36-6A87-4423-BC89-AF78F770F2A6@redhat.com> Message-ID: Yes thanks a lot ! I tried end-to-end (new ManageIQ master, new Services master and new agent (1.0.CR1)) in OpenShift, it worked for what I did. Thomas On Fri, Apr 28, 2017 at 9:45 AM, Heiko W.Rupp wrote: > Hey, > > the changes have now also been merged into ManageIQ master, > so we are on inventory.v3 end-to-end > > Joel has started a tech document for the Hawkular.org pages > you can find the PR here: > https://github.com/hawkular/hawkular.github.io/pull/311 > > Thanks > Heiko > > On 27 Apr 2017, at 10:04, Heiko W.Rupp wrote: > > > Hey, > > > > sorry for not sending this email earlier. > > > > Hawkular-services has in master and also release 0.36 > > switched to new Inventory.v3, which is Inventory on Metrics. > > The old Inventory component (that uses a relational DB) > > has been removed. > > The Ruby client gem has been bumped to version 3.0.1 > > to cater for this change [1]. > > Agent(s) 1.0.cr1 are also using this new Inventory code. > > > > Joel is currently preparing a show&tell to walk us through > > the new Inventory version and will also update docs on > > hawkular.org accordingly. > > > > My thanks goes to everyone involved to make this happen > > and especially to Joel who did the majority of the work. > > > > Heiko > > > > [1] And HawkFX in master is also using that gem version now. > > -- > > Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, > > Werner-von-Siemens-Ring 14, D-85630 Grasbrunn > > Handelsregister: Amtsgericht M?nchen HRB 153243 > > Gesch?ftsf?hrer: Charles Cachera, Michael Cunningham, Michael > > O'Neill, > > Eric Shander > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > -- > Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, > Werner-von-Siemens-Ring 14, D-85630 Grasbrunn > Handelsregister: Amtsgericht M?nchen HRB 153243 > Gesch?ftsf?hrer: Charles Cachera, Michael Cunningham, Michael O'Neill, > Eric Shander > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170428/fe2749c1/attachment.html From jshaughn at redhat.com Fri Apr 28 07:57:04 2017 From: jshaughn at redhat.com (Jay Shaughnessy) Date: Fri, 28 Apr 2017 07:57:04 -0400 Subject: [Hawkular-dev] HEADS UP: Hawkular Services is now on Inventory.v3 In-Reply-To: References: <8B112884-405D-433D-83F8-F111E1955C5E@redhat.com> <11F3ED36-6A87-4423-BC89-AF78F770F2A6@redhat.com> Message-ID: <3872f079-4b86-5d41-1c59-46624afaa176@redhat.com> Note that avail backfill is not working in services 0.36 but is fixed for 0.37. So stopping an agent will not successfully set its resources availabilities to down. On 4/28/2017 5:39 AM, Thomas Heute wrote: > Yes thanks a lot ! > > I tried end-to-end (new ManageIQ master, new Services master and new > agent (1.0.CR1)) in OpenShift, it worked for what I did. > > Thomas > > On Fri, Apr 28, 2017 at 9:45 AM, Heiko W.Rupp > wrote: > > Hey, > > the changes have now also been merged into ManageIQ master, > so we are on inventory.v3 end-to-end > > Joel has started a tech document for the Hawkular.org pages > you can find the PR here: > https://github.com/hawkular/hawkular.github.io/pull/311 > > > Thanks > Heiko > > On 27 Apr 2017, at 10:04, Heiko W.Rupp wrote: > > > Hey, > > > > sorry for not sending this email earlier. > > > > Hawkular-services has in master and also release 0.36 > > switched to new Inventory.v3, which is Inventory on Metrics. > > The old Inventory component (that uses a relational DB) > > has been removed. > > The Ruby client gem has been bumped to version 3.0.1 > > to cater for this change [1]. > > Agent(s) 1.0.cr1 are also using this new Inventory code. > > > > Joel is currently preparing a show&tell to walk us through > > the new Inventory version and will also update docs on > > hawkular.org accordingly. > > > > My thanks goes to everyone involved to make this happen > > and especially to Joel who did the majority of the work. > > > > Heiko > > > > [1] And HawkFX in master is also using that gem version now. > > -- > > Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, > > Werner-von-Siemens-Ring 14, D-85630 Grasbrunn > > Handelsregister: Amtsgericht M?nchen HRB 153243 > > Gesch?ftsf?hrer: Charles Cachera, Michael Cunningham, Michael > > O'Neill, > > Eric Shander > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > -- > Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, > Werner-von-Siemens-Ring 14, D-85630 Grasbrunn > Handelsregister: Amtsgericht M?nchen HRB 153243 > Gesch?ftsf?hrer: Charles Cachera, Michael Cunningham, Michael O'Neill, > Eric Shander > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170428/e635b26c/attachment.html From jtakvori at redhat.com Fri Apr 28 09:54:21 2017 From: jtakvori at redhat.com (Joel Takvorian) Date: Fri, 28 Apr 2017 15:54:21 +0200 Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: <1623540796.3765173.1493323793895.JavaMail.zimbra@redhat.com> References: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> <384747a5-79f1-fa4d-8bee-fb191212dc44@redhat.com> <476709290.3165329.1493237550085.JavaMail.zimbra@redhat.com> <1623540796.3765173.1493323793895.JavaMail.zimbra@redhat.com> Message-ID: What kind of functionality should be added to the golang client? Both for reading and writing inventory? FYI I'm currently learning a bit of golang and taking the online go tour On Thu, Apr 27, 2017 at 10:09 PM, John Mazzitelli wrote: > > but metrics that aren't consumed quickly should likely be discarded. > > Yes, I think we rely on HOSA to collect metrics on its intervals, not HJA. > HJA will just maintain the last known metric data point for each metric. > > > HJA would not have any ability to push to HOSA, right? So changes in > inventory would likely have a [short] delay > > before being pulled by HOSA. > > Yes, good point. HJA does not push to HOSA. > _______________________________________________ > hawkular-dev mailing list > hawkular-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170428/57487aa4/attachment.html From mazz at redhat.com Fri Apr 28 09:56:13 2017 From: mazz at redhat.com (John Mazzitelli) Date: Fri, 28 Apr 2017 09:56:13 -0400 (EDT) Subject: [Hawkular-dev] plan for HOSA to store inventory collected by Java Agent In-Reply-To: References: <112283139.3079811.1493224508406.JavaMail.zimbra@redhat.com> <384747a5-79f1-fa4d-8bee-fb191212dc44@redhat.com> <476709290.3165329.1493237550085.JavaMail.zimbra@redhat.com> <1623540796.3765173.1493323793895.JavaMail.zimbra@redhat.com> Message-ID: <1983746678.4238920.1493387773748.JavaMail.zimbra@redhat.com> > What kind of functionality should be added to the golang client? Both for > reading and writing inventory? Yes. > > FYI I'm currently learning a bit of golang and taking the online go tour > > On Thu, Apr 27, 2017 at 10:09 PM, John Mazzitelli wrote: > > > > but metrics that aren't consumed quickly should likely be discarded. > > > > Yes, I think we rely on HOSA to collect metrics on its intervals, not HJA. > > HJA will just maintain the last known metric data point for each metric. > > > > > HJA would not have any ability to push to HOSA, right? So changes in > > inventory would likely have a [short] delay > > > before being pulled by HOSA. > > > > Yes, good point. HJA does not push to HOSA. > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > From hrupp at redhat.com Fri Apr 28 11:18:16 2017 From: hrupp at redhat.com (Heiko W.Rupp) Date: Fri, 28 Apr 2017 17:18:16 +0200 Subject: [Hawkular-dev] Some food for thought about improving the release of (large) features Message-ID: Hey, some of us just had a meeting to recapture parts of the switch from Inventory.v2 to .v3, where things went less easy (on the java side) than I expected. We identified a few areas where we could improve: - Timeouts. Some tests were failing on local machines but not on travis (and we had seen that in the direction in the past as well). We need to be better at not assuming timing, as we can't know timing in the target environments as well. Similarly the test against live server was waiting 500*a few seconds until inventory(.old) came up. Some waiting is good, but the question is if e.g. inventory does not come up after some reasonable time, if we should not abort the test as this may show real issues. - Test reliability (the above is part of this). We need to try to have more unit and also integration tests and make them more reliable. During the merge we saw test failures on developer machines while Travis was good. It turned out that this was due to timing. In the (RHQ) past we saw test failures because of test ordering. We should perhaps try to make our (integration) tests in random order on purpose, as in reality, the user will not run the code in the order we assume in tests either (yes, that may make setup and tear-down more complex). - Making tests more end-to-end. Right now we have no idea (from the java side) about the consequences of e.g. renaming a resource in the agent to the display of this resource in ManageIQ. Luckily we already have the ruby-gem tests that run against the live server. Perhaps we can extend this somehow into MiQ test suite, so that this also tests against latest hawkular-services master. Or record some interactions of MiQ with H-services via the gem and have those interactions be re-played against the live server (there will be a need for placeholders, but that is something that cassettes already support) - Way of working for such all-over changes: We were talking that in this case it could be good to do that in a series of feature branches which can use src-deps so that the feature branches all applied give the desired new state. And only if all that is good, send pull-requests and apply them to merge the full stream of work into master and get releases of the components out. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170428/8316aac0/attachment-0001.html