Cross-Tenant endpoints in Alerting on OS
by Jay Shaughnessy
On 2/23/2017 6:05 PM, Matt Wringe wrote:
> Is there any reason why this being sent in private emails and not to a mailing list?
Matt, Not really, sending to dev-list for anyone interested in the
discussion...
> ----- Original Message -----
>> There was an IRC discussion today about $SUBJECT. Here is a summary of
>> a conversation Matt and I had to drill down into whether there was a
>> cross-tenant security concern with the Alerting API in OS. In short,
>> the answer seems to be no. Alerting (1.4+) offers two endpoints for
>> fetching cross-tenant: /alerts/admin/alerts and /alerts/admin/events.
>> Note that the 'admin' is just in the path, and was chosen just to group
>> what we deemed were admin-level endpoints, the first two of which are
>> these cross-tenant fetches. The 'admin' does not mean anything else in
>> this context, it does not reflect a special user or tenant. The way
>> these endpoints work is that that they accept a Hawkular-Tenant HTTP
>> header that can be a comma-separated-list of tenantIds. As with any of
>> the alerting endpoints. Alerting does not perform any security in the
>> request handling. But, in OS the HAM deployments both have the OS
>> security filtering in place. That filtering does two things, for a
>> cluster-admin user it's basically a pass-thru, the CSL Hawkular-Tenant
>> header is passed on and the endpoints work. For all other users the
>> Hawkular-Tenant header is validated. Because each project name is a
>> tenant name, the value must match a project name. As such, the
>> validation fails if a CSL is supplied. This is decent behavior for now
>> as it prevents any undesired access. Note that as a corner-case, these
>> endpoints will work fine if the header just supplies a single tenant, in
>> which case they are basically the same as the typical single-tenant
>> fetch endpoints.
> What has happened is now Alerts is not considering the Hawkular-tenant header to contain just a string, but a comma separated lists of strings.
>
> eg "Hawkular-tenant: projectA,projectB"
Note, not in general, comma-separated-lists handled only for the two
cross-tenant endpoints mentioned above.
> The OpenShift filter still considers this to be a string, so it will check with OpenShift if the user has permission to access the project named with a string value of "projectA,projectB". Since a project cannot have a ',' within its name, this check will always fail and return an access denied error.
>
> If the user is a cluster level user they are given access to everything, even impossibly named projects. So a cluster level user will happen to be able to use the current setup just due to how this works.
>
> So there doesn't appear to be any security issue that we need to deal with immediately, but we do probably want to handle this properly in the future. It might not be too difficult to add support to the tenant to consider a csl.
>
>> I'm not totally familiar with the Metrics approach to cross-tenant
>> handling but going forward we (Metrics and Alerting) should probably
>> look for some consistency, if possible. Moreover, any solution should
>> reflect what best serves OS. The idea of a CSL for the header is fairly
>> simple and flexible. It may be something to consider, for the OS filter
>> it would mean validating that the bearer has access to each of the
>> individual tenants before forwarding the request.
> I don't recall any meetings about adding multitenancy to Metrics. From what I recall, there is no plans at all to introduce multitenancy at all for metrics.
>
> If I was aware of this discussion when this was brought up for alerts, I would have probably objected to the endpoint being called 'admin' since I don't think that reflects what the true purpose of this is suppose to be. Its not really an admin endpoint, but an endpoint for cross-tenancy. I could have access to projectA and projectB, but not be an 'admin'
>
> If we are making changes like this which affect security, I would really like to be notified so that I can make sure our security filters will function properly. Even if I am in the meeting when its being discussed it would be good to ping me on the PR with the actual implementation.
Of course. This stuff went in in mid November and at that time we (in
alerting) were really just getting settled with the initial integration
into metrics for OS. Going forward I think we have a better idea of
what is relevant to OS and can more easily flag items of import.
7 years, 10 months
Proposing closer integration of APM and "Hawkular metrics" on Kubernetes / OpenShift
by Heiko W.Rupp
Hi,
right now Hawkular metrics and Hawkular APM are going
relatively separate ways. This is in part due to the backend choice,
but probably also for other reasons.
I am proposing that we try to get the two closer together because at the
end neither tracing data alone, not classic monitoring data can answer
all the questions like:
APM
- why is my service XY slow (my be overload of underlying CPU)
- how much disk will my service need in two years
- how much network usage did my service have yesterday
Classic montoring
- which service will fail if I pull the plug here
- what are customers buying
- why is my service slow (may be come from a dependency)
I am proposing that we integrate the two over the UI - in the first
scenario
here the key driver is the APM UI with its trace diagrams (red boxes).
A klick on such a box will then show related metrics from the
classic monitoring.
On the level of the individual pod, both APM and Classic
'instrumentations'
are present. For JVM-based apps this is on one side the APM agent and/or
APM instrumentation ("OT-instrumentation") (*a) On the other side the
jolokia agent/agent bond (*b)
In this first scenario, APM and classic still have separate agents and
connections to the
backends and different backend storage.
The 2nd scenario, assumes that it is possible to use only one agent
binary that
does both APM and classic metric export. For classic metrics, Hosa will
poll it with
P8s metrics. And on top APM trace data will also be made available for
grab by
Hosa, which will then forward them to the APM server.
Thoughts?
Heiko
*a) I propose to always deploy the APM agent to get a quick and easy
coverage
of standard scenarios, so that the user only needs explicit
instrumentation to
increase granularity and/or to cover cases the agent can't cover.
Also "manual" instrumentation should be able to use the agent's
connection to
talk to the APM server.
*b) I think it would make sense to always use the Prometheus protocol
(and Hosa
may learn how to use the more efficient binary protocol) as Jolokia/http
is JVM/Jmx
specific, while P8s exporters also exist for other environments like
Node or Ruby
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
7 years, 10 months
more openshift issues
by John Mazzitelli
If I start openshift with "sudo ./openshift start" and then try to log in like this:
oc login -u system:admin
What would cause this:
Authentication required for https://192.168.1.15:8443 (openshift)
Username: system:admin
Password:
error: username system:admin is invalid for basic auth
When I start with "oc cluster up" I do not get asked for a password and it "just works"
7 years, 10 months
openshift - using cluster up but building from source
by John Mazzitelli
Has anyone been able to use "oc cluster up --metrics" in order to run OpenShift Origin *and* Origin Metrics but running a local build (i.e. I need to pick up changes in master branch of Origin/Origin Metrics that aren't released yet).
The docs make it look very complicated, and nothing I found seems to help a dev get this up and running quickly without having to look at tons of docs and run lots of commands with bunches of yaml :).
I'm hoping it is easy, but not documented.
This link doesn't even mention "cluster up" let alone running with Origin Metrics: https://github.com/openshift/origin/blob/master/CONTRIBUTING.adoc#develop...
If I run "openshift start" - how do I get my own build of Origin Metrics to deploy, like "oc cluster up --metrics" does?
It seems no matter what I do, using "cluster up" pulls down images from docker hub. I have no idea how to run Origin+Metrics using a local build.
I'm hoping someone knows how to do this and can give me the steps.
7 years, 10 months
Re: [Hawkular-dev] Test Account credentials for Hawkular Android Client
by Anuj Garg
Hello pawan. I was last maintainer of this android client. Lets talk on
hangout for detailed interaction if you interested in maintaining this code
On 21 Feb 2017 1:52 p.m., "Thomas Heute" <theute(a)redhat.com> wrote:
Then it would be localhost:8080 with myUsername and myPassword as defined
in step 3 of the guide.
On Tue, Feb 21, 2017 at 9:06 AM, Pawan Pal <pawanpal004(a)gmail.com> wrote:
> Hi,
> I set up my Hawkular server following this guide :
> http://www.hawkular.org/hawkular-services/docs/installation-guide/
>
>
>
>
> On Tue, Feb 21, 2017 at 11:37 AM, Pawan Pal <pawanpal004(a)gmail.com> wrote:
>
>> Hi all,
>> I would like to know the credentials of any testing account for Hawkular
>> android-client. I found jdoe/ password, but it is not working. Also please
>> give server and port.
>>
>> Thanks.
>>
>> --
>> Pawan Pal
>> *B.Tech (Information Technology and Mathematical Innovation)*
>> *Cluster Innovation Centre, University of Delhi*
>>
>>
>>
>>
>
>
> --
> Pawan Pal
> *B.Tech (Information Technology and Mathematical Innovation)*
> *Cluster Innovation Centre, University of Delhi*
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
7 years, 10 months
Test Account credentials for Hawkular Android Client
by Pawan Pal
Hi all,
I would like to know the credentials of any testing account for Hawkular
android-client. I found jdoe/ password, but it is not working. Also please
give server and port.
Thanks.
--
Pawan Pal
*B.Tech (Information Technology and Mathematical Innovation)*
*Cluster Innovation Centre, University of Delhi*
7 years, 10 months
RxJava2 preliminary testing
by Michael Burman
Hi,
I did yesterday evening and today some testing on how using RxJava2
would benefit us (I'm expecting more from RxJava 2.1 actually, since it
has some enhanced parallelism features which we might benefit from).
Short notes from RxJava2 migration, it's more painful than I assumed.
The code changes can be small in terms of lines of code changed, but
almost every method has had their signature or behavior changed. So at
least I've had to read the documentation all the time when doing things
and trying to unlearn what I've done in the RxJava1.
And all this comes with a backwards compatibility pressure for Java 6
(so you can't benefit from many Java 8 advantages). Reactive-Commons /
Reactor have started from Java 8 to provide cleaner implementation. Grr.
I wrote a simple write path modification in the PR #762 (metrics) that
writes Gauges using RxJava2 ported micro-batching feature. There's still
some RxJavaInterOp use in it, so that might slow down the performance a
little bit. However, it is possible to merge these two codes. There are
also some other optimizations I think could be worth it.
I'd advice against it though, reading gets quite complex. I would almost
suggest that we would do the MetricsServiceImpl/DataAccessImpl merging
by rewriting small parts at a time in the new class with RxJava2 and
make that call the old code with RxJavaInterOp. That way we could move
slowly to the newer codebase.
I fixed the JMH-benchmarks (as they're not compiled in our CI and were
actually broken by some other PRs) and ran some tests. These are the
tests that measure only the metrics-core-service performance and do not
touch the REST-interface (or Wildfly) at all, thus giving better
comparison in how our internal changes behave.
What I'm seeing is around 20-30% difference in performance when writing
gauges this way. So this should offset some of the issues we saw when we
improved error handling (which caused performance degradation). I did
ran into the HWKMETRICS-542 (BusyPoolException) so the tests were run
with 1024 connections.
I'll continue next week some more testing, but at the same time I proved
that the micro-batching features do improve performance in the internal
processing, especially when there's small amount of writers to a single
node. But testing those features could probably benefit from more
benchmark tests without WIldfly (which takes so much processing power
that most performance improvements can't be measured correctly anymore).
- Micke
7 years, 10 months
HOSA and conversion from prometheus to hawkular metrics
by John Mazzitelli
The past several days I've been working on an enhancement to HOSA that came in from the community (in fact, I would consider it a bug). I'm about ready to merge the PR [1] for this and do a HOSA 1.1.0.Final release. I wanted to post this to announce it and see if there is any feedback, too.
Today, HOSA collects metrics from any Prometheus endpoint which you declare - example:
metrics
- name: go_memstats_sys_bytes
- name: process_max_fds
- name: process_open_fds
But if a Prometheus metric has labels, Prometheus itself considers each metric with a unique combination of labels as an individual time series metric. This is different than how Hawkular Metric works - each Hawkular Metric metric ID (even if its metric definition or its datapoints have tags) is a single time series metric. We need to account for this difference. For example, if our agent is configured with:
metrics:
- name: jvm_memory_pool_bytes_committed
And the Prometheus endpoint emits that metric with a label called "pool" like this:
jvm_memory_pool_bytes_committed{pool="Code Cache",} 2.7787264E7
jvm_memory_pool_bytes_committed{pool="PS Eden Space",} 2.3068672E7
then to Prometheus this is actually 2 time series metrics (the number of bytes committed per pool type), not 1. Even though the metric name is the same (what Prometheus calls a "metric family name"), there are two unique combinations of labels - one with "Code Cache" and one with "PS Eden Space" - so they are 2 distinct time series metric data.
Today, the agent only creates a single Hawkular-Metric in this case, with each datapoint tagged with those Prometheus labels on the appropriate data point. But we don't want to aggregate them like that since we lose the granularity that the Prometheus endpoint gives us (that is, the number of bytes committed in each pool type). I will say I think we might be able to get that granularity back through datapoint tag queries in Hawkular-Metrics but I don't know how well (if at all) that is supported and how efficient such queries would be even if supported, and how efficient storage of these metrics would be if we tag every data point with these labels (not sure if that is the general purpose of tags in H-Metrics). But, regardless, the fact that these really are different time series metrics should (IMO) be represented as different time series metrics (via metric definitions/metric IDs) in Hawkular-Metrics.
To support labeled Prometheus endpoint data like this, the agent needs to split this one named metric into N Hawkular-Metrics metrics (where N is the number of unique label combinations for that named metric). So even though the agent is configured with the one metric "jvm_memory_pool_bytes_committed" we need to actually create two Hawkular-Metric metric definitions (with two different and unique metric IDs obviously).
The PR [1] that is ready to go does this. By default it will create multiple metric definitions/metric IDs in the form "metric-family-name{labelName1=labelValue1,labelName2=labelValue2,...}" unless you want a different form in which case you can define an "id" and put in "${labelName}" in the ID you declare (such as "${oneLabelName}_my_own_metric_name_${theOtherLabelName}" or whatever). But I suspect the default format will be what most people want and thus nothing needs to be done. In the above example, two metric definitions with the following IDs are created:
1. jvm_memory_pool_bytes_committed{pool=Code Cache}
2. jvm_memory_pool_bytes_committed{pool=PS Eden Space}
--John Mazz
[1] https://github.com/hawkular/hawkular-openshift-agent/pull/117
7 years, 10 months
Collecting PV usage ?
by Thomas Heute
Mazz,
in your metric collection adventure for HOSA have you met a way to see the
usage of PVs attached to a pod ?
User should know (be able to visualize) how much of the PVs are used and
then be alerted if it reach a certain %.
Thomas
7 years, 10 months
