Some food for thought about improving the release of (large) features
by Heiko W.Rupp
Hey,
some of us just had a meeting to recapture parts of the switch from
Inventory.v2 to .v3, where things went less easy (on the java side) than
I expected.
We identified a few areas where we could improve:
- Timeouts. Some tests were failing on local machines but not on travis
(and we had seen that in the direction in the past as well). We need to
be better at not assuming timing, as we can't know timing in the target
environments as well.
Similarly the test against live server was waiting 500*a few seconds
until inventory(.old) came up. Some waiting is good, but the question is
if e.g. inventory does not come up after some reasonable time, if we
should not abort the test as this may show real issues.
- Test reliability (the above is part of this). We need to try to have
more unit and also integration tests and make them more reliable. During
the merge we saw test failures on developer machines while Travis was
good. It turned out that this was due to timing. In the (RHQ) past we
saw test failures because of test ordering. We should perhaps try to
make our (integration) tests in random order on purpose, as in reality,
the user will not run the code in the order we assume in tests either
(yes, that may make setup and tear-down more complex).
- Making tests more end-to-end. Right now we have no idea (from the java
side) about the consequences of e.g. renaming a resource in the agent to
the display of this resource in ManageIQ. Luckily we already have the
ruby-gem tests that run against the live server. Perhaps we can extend
this somehow into MiQ test suite, so that this also tests against latest
hawkular-services master. Or record some interactions of MiQ with
H-services via the gem and have those interactions be re-played against
the live server (there will be a need for placeholders, but that is
something that cassettes already support)
- Way of working for such all-over changes: We were talking that in this
case it could be good to do that in a series of feature branches which
can use src-deps so that the feature branches all applied give the
desired new state. And only if all that is good, send pull-requests and
apply them to merge the full stream of work into master and get releases
of the components out.
8 years, 11 months
plan for HOSA to store inventory collected by Java Agent
by John Mazzitelli
The first use-case for needing HOSA to collect and store inventory has come in - from the Fuse team, specifically. This email is going to explain the current plan to get this to happen. Feel free to chime in with thoughts.
Right now, HOSA does not support collecting and storing inventory. HJA (and HWFA) supports collecting and storing inventory along with metrics.
HOSA has the privileges necessary to write directly to Origin Metrics. HJA does not.
We need to combine these to get what we need (and we need this soon because Fuse needs it).
Currently, the idea is:
a) HJA will have a new storage adapter mode "HOSA" (right now it has "HAWKULAR" and "METRICS" - the former is to support running with a full hawkular server such as when running within CloudForms and the latter is for when running with just a H-Metrics server). With this new HOSA mode, HJA will not store metrics or inventory directly into H-Metrics. Instead it will "cache" the data and wait for HOSA to come and ask for the data.
b) HOSA will have a new endpoint type "hawkular" (right now it has "prometheus" and "jolokia" and "json"). This new type will tell HOSA to read data from some HJA (which will including inventory data) and HOSA will simply pass that data on to Origin Metrics. HOSA will read both metrics and inventory from HJA and HOSA will store that data directly to Origin Metrics essentially making HOSA a proxy to Origin Metrics.
I don't know if HOSA will decorate the data with its own metadata (like tags and things) or if it will truly be a pass-through. I suspect HOSA will need to massage the data it gets from HJA to ensure the data IDs are unique across the OpenShift cluster and things like that.
That's the overall idea. I'm sure there are going to be stumbling blocks along the way that is going to force us to change some things around. But at a high level, I think it should work. HJA caches inventory and metrics rather than writing directly to an H-Metrics server - and HOSA will collect that cached data in HJA and it will be the one to store it in Origin Metrics.
8 years, 11 months
HEADS UP: Hawkular Services is now on Inventory.v3
by Heiko W.Rupp
Hey,
sorry for not sending this email earlier.
Hawkular-services has in master and also release 0.36
switched to new Inventory.v3, which is Inventory on Metrics.
The old Inventory component (that uses a relational DB)
has been removed.
The Ruby client gem has been bumped to version 3.0.1
to cater for this change [1].
Agent(s) 1.0.cr1 are also using this new Inventory code.
Joel is currently preparing a show&tell to walk us through
the new Inventory version and will also update docs on
hawkular.org accordingly.
My thanks goes to everyone involved to make this happen
and especially to Joel who did the majority of the work.
Heiko
[1] And HawkFX in master is also using that gem version now.
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
8 years, 11 months
bug in MiQ / javaagent integration - needs to be fixed
by John Mazzitelli
There is a problem with the new Java agent and MiQ.
This problem is because the agent is no longer in WildFly's DMR tree of resources (because only WildFly subsystems are in the DMR tree).
Look here:
https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent...
So when the agent (java or subsystem agent, doesn't matter) is monitoring EAP over the DMR management interface, that's the metadata it uses. You can see we inject two non-WildFly attributes (immutable and in-container) under the WildFly Server type. But these come from the agent - hence why you see:
path: /subsystem=hawkular-wildfly-agent
But when running this as a java agent, that resource doesn't exist - there is no agent subsystem anymore. So the immutable and in-container attributes end up not getting defined under the WildFly Server resource.
The problem is MiQ looks for those attributes under WildFly Servers to determine if the agent is in immutable mode and if its running in a container.
In effect, we are trying to inject under a WildFly Server resource completely unrelated information (immutable and in-container attributes) - those are related to the AGENT, not the WildFly Server itself (i.e. you won't see those two attributes if you use the jboss-cli under the root "/" resource).
We probably should have had MiQ examine the AGENT resource for this (since "immutable" and "in container" are attributes on the agent, not the server). And we shouldn't define "immutable" and "in container" under the WildFly Server resource because that is depending on the fact that the agent is running as a subsystem. As a Java Agent, as you see, those attributes will never show any values because there is no agent subsystem.
MiQ can look at the agent attributes because the agent itself exposes immutable and in-container (because they ARE actual properties the agent has) in inventory. Under the agent resource in inventory you will see these:
https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent...
Note that we DO expose immutable and in-container under the Agent JMX resource in inventory also - and it has the same resource name as the subsystem agent (the name as you will see it in inventory):
https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent...
Hence why I think the best/easiest thing to do is get MiQ to examine the attributes on the resource where they really belong (and we should remove them from the WildFly Server type).
8 years, 11 months
OpenShift OAuth authentication and authorization for Hawkular APM
by Lars Milland
Hi
It would be really great if a functionality for Hawkular APM could be
found/established, matching the one that exists for Hawkular Metrics wise
for OpenShift, where the metrics are stored per tenant/namespace, and then
Hawkular security wise is integrated to the OAuth based security model of
OpenShift.
Is that a requirement/feature that have been considered? Or would it maybe
already be possible to integrate the Hawkular APM components to OpenShift
OAuth based security. Even if the Hawkular APM storage and security model
would not fit to the fully multitenant way of OpenShift, if just the
security model of a Hawkular APM installation could be connected to the
OpenShift OAuth model, then one Hawkular APM instance could be setup with
"service account tokens" used for sending metrics to the instance, and users
could log into the Hawkular APM UI with again OpenShift OAuth managed
credentials, mapped to roles coming from the OAuth ticket. Much the same way
that the security model of the OpenShift integrated Jenkins works - see:
https://github.com/openshift/jenkins-openshift-login-plugin
The current security model of APM is rather limited as far as I understand -
and based solely on a single manually fixed username/password for both
contributing application performance metrics/log entries, and same for the
Hawkular APM UI.
Best regards
Lars Milland
8 years, 11 months
Add functionality to change password.
by Mohammad Murad
Hello
I'm a contributor to the Android Client of Hawkular.
On the Gitter channel I suggested that we should give the user an option to
change the password. Currently there is no REST API for that. This will be
very helpful if the credentials of the user are compromised.
Heiko W. Rupp asked me suggest this here.
If I can help with this, please let me know.
Regards
M. Murad
GitHub <http://github.com/free4murad>
8 years, 12 months
Hawkular Agent 1.0.0.CR1 has been released - inventory in metrics
by John Mazzitelli
Hawkular Agent 1.0.0.CR1 has been released.
This includes the new "inventory in metrics" feature - Hawkular-Inventory is no longer used to store inventory, the inventory is now stored in Hawkular-Metrics within Cassandra.
This is a *significant* change and needs people to beat on it heavily before we can claim victory (hence why it has the CR1 designation and not Final).
So, please grab it and use it when you need to use an agent.
If you find any bugs, please submit HWKAGENT JIRAs at https://issues.jboss.org/projects/HWKAGENT
--John Mazz
[this message was sent on April 24, 2017 at 9:19pm EDT]
8 years, 12 months
Proposal for Query DSL
by Benjamin Heiskell
Hey all,
I recently deployed Hawkular because I found its scaling model appealing.
It has been working well for our basic monitoring usage when combined with
Grafana.
One of my consumers is trying to graph KPIs that involve cross-metric
mathematics like sums, division, and moving averages (i.e., post-ingress
transformations). I looked into how to accomplish this with Hawkular, but I
didn't see any obvious way to do it. The stats endpoint seems to only
support aggregations on a single metric/tag at a time. I'm finding that we
can't visualize these metrics the same way you can with other TSDBs like
Graphite/Prometheus.
I did a little research, and found Heiko Rupp's blog post on HawkFX where
he seems to already be exploring this idea [1]. Perhaps the query language
item on the 2017 road map refers to incorporating this?
I would like to assist in developing this feature. Given that there is
already a tags DSL, it seems like it would make sense to extend that to
support metric transformations.
Grafana is also considering adding this to Grafana itself, but it doesn't
look like the ticket has any traction [2]. Additionally, shipping
potentially megabytes of data to Grafana (or the user's browser) to
aggregate doesn't seem ideal.
I documented a few TSDB DSLs styles here [3].
What do you all think?
Thanks,
Ben
[1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html
[2] https://github.com/grafana/grafana/issues/3677
[3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3
9 years
OpenShift - An error occurred getting metrics.
by Anton
Hello
I have tried to run Hawkular in both `oc cluster up` and OpenShift,
different versions of both.
The documentation for `oc cluster up` says (
https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md#i...)
that by using the `--metrics` parameter, metrics is installed.
Its a similar story when installing OpenShift using Openshift-ansible.
However, in both cases, when logging into the OpenShift webconsole I see an
error message saying:
An error occurred getting metrics. Open Metrics URL | Don't Show Me Again
It would seem there is more to correctly installing Hawkular metrics than
just using the parameter.
What else needs to be done? And is it possible get the additional info
added to the documentation for openshift-ansible and `oc cluster up`?
Thanks
9 years
