[Hawkular-dev] OpenShift agent - multiple identity for certs

Wed Dec 28 09:29:49 EST 2016

OK, yes, its the mutual auth that I was missing. This is why you need to use that specific client cert, because I assume the server-side will only trust that client cert and no others (unless you somehow add your generated cert to some kind of "trust store" so the server trusts it as well).

Hmm... so this is an interesting problem. For now, I would say the answer is "generate a cert for your agent, and tell your servers to trust it" if that is even possible but even so I assume that isn't a nice way to do it??

The problem with the agent is the endpoints are defined by your pods (they are config maps on the projects where the pods are deployed) and the people doing those pod deployments are typically not the same admin person who configures and sets up the agent. So how would the pod configmap authors know the locations and names of the key-pairs or have privileges to install those keys on the agent (I'm thinking of the use case where OpenShift is running and developers/deployers are deploying their own applications and want to collect their own app metrics).

Anyway, I created an issue in github where we can discuss this further there:

https://github.com/hawkular/hawkular-openshift-agent/issues/75

----- Original Message -----
> Hi John,
> 
> Below are the steps i am doing;
> 
> 1. The certs for etcd are here (see path below), both server and client. I
> am only interested in the client for the agent:
> 
> [root at localhost master]# pwd
> /var/lib/origin/openshift.local.config/master
> 
> [root at localhost master]# ls -ltr *etc*
> -rw-rw-rw-. 1 root root 1078 Sep 23 18:31 master.etcd-client.crt
> -rw-rw-rw-. 1 root root 1679 Sep 23 18:31 master.etcd-client.key
> -rw-rw-rw-. 1 root root 1675 Sep 23 18:31 etcd.server.key
> -rw-rw-rw-. 1 root root 2550 Sep 23 18:31 etcd.server.crt
> 
> 2. I add the client certs as secrets and mount them to the agent:
> 
> oc project openshift-infra
> oc secrets new etcd-client-crt master.etcd-client.crt
> oc secrets new etcd-client-key master.etcd-client.key
> 
> oc volume rc/hawkular-openshift-agent --add --name=etcd-client-crt
> --type=secret --secret-name=etcd-client-crt
> --mount-path=/run/secrets/etcd-client-crt
> oc volume rc/hawkular-openshift-agent --add --name=etcd-client-key
> --type=secret --secret-name=etcd-client-key
> --mount-path=/run/secrets/etcd-client-key
> 
> 3. Then edit the config map of the agent and add in the below, which
> matches the above secret mounts:
> 
> 
> oc edit configmap hawkular-openshift-agent-configuration
> 
> identity:
>   cert_file: /run/secrets/etcd-client-crt/master.etcd-client.crt
>   private_key_file: /run/secrets/etcd-client-key/master.etcd-client.key
> 
> 4. Restart the pod to force a refresh and check the logs, which shows:
> 
> 
> I1228 10:20:18.799687       1 prometheus_metrics_collector.go:97] DEBUG:
> Told to collect [2] Prometheus metrics from [https://172.17.0.8:9779/metrics
> ]
> I1228 10:20:18.984615       1 metrics_storage.go:152] DEBUG: Stored
> datapoints for [2] metrics
> 
> I now have a working agent collecting from etcd.
> 
> Since etcd is mutual auth - maybe thats what is causing the confusion, as
> you keep mentioning server-certs - i am not sure how generating my own
> client certs helps.
> 
> 
> But since you said try it, i have with the below commands but as i would
> expect, got a TLS error:
> 
> openssl req -newkey rsa:2048 -nodes -keyout agent.key -out agent.csr -subj
> "/C=UK/ST=Yorkshire/L=Leeds/O=Home/CN=hawkular-agent"
> openssl x509 -signkey agent.key -in agent.csr -req -days 365 -out agent.crt
> 
> Just incase there are cross wires; etcd in OCP requires mutual auth (thats
> how i understand it), so thats the reason i am adding in the etcd client
> certs to the identity section of the agent configmap. If i needed to
> monitor another endpoint which was also mutual auth, with the current setup
> i wouldn't be able to do that.
> 
> If theres anything you want me to try, happy to do so.
> 
> Cheers.
> 
> On Sun, Dec 25, 2016 at 1:23 PM, John Mazzitelli <mazz at redhat.com> wrote:
> 
> > Gareth,
> >
> > OK, there are a couple things here that I'm confused about. This is how I
> > would understand things working.
> >
> > If you want to connect to any https endpoint, the agent will need SOME
> > identity (so you have to give it SOME public/private key pair - which is
> > what the Identity section does). It doesn't have to be the key-pair of the
> > server (in fact, under normal situations it is not - the server is
> > identified with its own public/private key and the client with another).
> > But the point is, if you are connecting to an https endpoint, you can't
> > leave Identity section out of the agent config.
> >
> > So when you say, "without the "Identity" configuration section set on the
> > agent config, i'd get a TLS error" this is what I would expect. You can't
> > leave the Identity section out when connecting via https because in that
> > case the agent has no keys to talk TLS to the server.
> >
> > What does your agent config look like when you get things to work? (I
> > assume you do get it to work because you said without the Identity you get
> > a TLS error, which implies you do get it to work WITH an Identity section -
> > is this correct?) What key files are you putting in the agent Identity when
> > you get it to work?
> >
> > So I guess what I am saying is - have you tried to generate your own
> > certificate and assigned it to your agent's Identity and then tried to
> > connect to multiple https endpoints? Because as I mentioned earlier in
> > another post, the agent today doesn't do server-cert verification, so it
> > should "just work". You shouldn't need different Identities per endpoint.
> > Once we add in verification, the endpoints you want to collect metrics from
> > would need their server-side certs to be signed with a CA that the agent
> > trusts (i.e. from the agent host's default root CA set) - we would then
> > have to add the ability for the agent to be told about different CAs in
> > case your server-side certs are, say, self-signed or signed with your own
> > CA that isn't a trusted one found in the host's default root CA set.
> >
> > Oh, and, Merry Christmas!
> >
> > John Mazz
> >
> > ----- Original Message -----
> > > One of the first services i am trying to monitor is etcd. etcd in OCP is
> > > configured as per the below:
> > >
> > > /var/lib/origin/openshift.local.config/master/master-config.yaml
> > >
> > >
> > > etcdClientInfo:
> > >   ca: ca.crt
> > >   certFile: master.etcd-client.crt
> > >   keyFile: master.etcd-client.key
> > >   urls:
> > >   - https://10.2.2.2:4001
> > >
> > > Which responds with the below cURL:
> > >
> > > curl https://10.2.2.2:4001/metrics --cacert ./ca.crt --cert
> > > ./master.etcd-client.crt --key ./master.etcd-client.key
> > >
> > > So without the "Identity" configuration section set on the agent config,
> > > i'd get a TLS error. As etcd is a core part of OCP, I don't have much
> > > control over the client certs and expect there might be other services
> > > which require the same setup using different certs that i might want to
> > > monitor.
> > >
> > > Hope that makes things clear, and Merry Christmas.
> > >
> > > Cheers.
> > >
> > > On Sat, Dec 24, 2016 at 3:30 PM, John Mazzitelli <mazz at redhat.com>
> > wrote:
> > >
> > > > > Currently it seems you can only provide the agent configmap with the
> > > > identity
> > > > > field. But what i want to actually do, is provide this based on the
> > pods
> > > > > config map>
> > > > > [chomp]
> > > > > Is that possible? or planned for the future?
> > > >
> > > > I was hoping this wasn't going to be needed :) But we did talk about
> > it.
> > > >
> > > > It is not possible today because there is one major problem with what
> > you
> > > > suggest that would need to be solved somehow:
> > > >
> > > > > cert_file: /var/run/secrets/client-crt/client.crt
> > > > > private_key_file: /var/run/secrets/client-key/client.key
> > > >
> > > > That is inside your configmap on your OpenShift project (which may or
> > may
> > > > not be the same project where the agent is deployed).
> > > >
> > > > So - what file system is that actually referring to? And how does the
> > > > agent get access to those files?
> > > > _______________________________________________
> > > > hawkular-dev mailing list
> > > > hawkular-dev at lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > > >
> > >
> >
> 