[Hawkular-dev] OpenShift agent - multiple identity for certs

[Gareth Healy]

Hi John,

Below are the steps i am doing;

1. The certs for etcd are here (see path below), both server and client. I
am only interested in the client for the agent:

[root at localhost master]# pwd
/var/lib/origin/openshift.local.config/master

[root at localhost master]# ls -ltr *etc*
-rw-rw-rw-. 1 root root 1078 Sep 23 18:31 master.etcd-client.crt
-rw-rw-rw-. 1 root root 1679 Sep 23 18:31 master.etcd-client.key
-rw-rw-rw-. 1 root root 1675 Sep 23 18:31 etcd.server.key
-rw-rw-rw-. 1 root root 2550 Sep 23 18:31 etcd.server.crt

2. I add the client certs as secrets and mount them to the agent:

oc project openshift-infra
oc secrets new etcd-client-crt master.etcd-client.crt
oc secrets new etcd-client-key master.etcd-client.key

oc volume rc/hawkular-openshift-agent --add --name=etcd-client-crt
--type=secret --secret-name=etcd-client-crt
--mount-path=/run/secrets/etcd-client-crt
oc volume rc/hawkular-openshift-agent --add --name=etcd-client-key
--type=secret --secret-name=etcd-client-key
--mount-path=/run/secrets/etcd-client-key

3. Then edit the config map of the agent and add in the below, which
matches the above secret mounts:


oc edit configmap hawkular-openshift-agent-configuration

identity:
  cert_file: /run/secrets/etcd-client-crt/master.etcd-client.crt
  private_key_file: /run/secrets/etcd-client-key/master.etcd-client.key

4. Restart the pod to force a refresh and check the logs, which shows:


I1228 10:20:18.799687       1 prometheus_metrics_collector.go:97] DEBUG:
Told to collect [2] Prometheus metrics from [https://172.17.0.8:9779/metrics
]
I1228 10:20:18.984615       1 metrics_storage.go:152] DEBUG: Stored
datapoints for [2] metrics

I now have a working agent collecting from etcd.

Since etcd is mutual auth - maybe thats what is causing the confusion, as
you keep mentioning server-certs - i am not sure how generating my own
client certs helps.


But since you said try it, i have with the below commands but as i would
expect, got a TLS error:

openssl req -newkey rsa:2048 -nodes -keyout agent.key -out agent.csr -subj
"/C=UK/ST=Yorkshire/L=Leeds/O=Home/CN=hawkular-agent"
openssl x509 -signkey agent.key -in agent.csr -req -days 365 -out agent.crt

Just incase there are cross wires; etcd in OCP requires mutual auth (thats
how i understand it), so thats the reason i am adding in the etcd client
certs to the identity section of the agent configmap. If i needed to
monitor another endpoint which was also mutual auth, with the current setup
i wouldn't be able to do that.

If theres anything you want me to try, happy to do so.

Cheers.

On Sun, Dec 25, 2016 at 1:23 PM, John Mazzitelli <mazz at redhat.com> wrote:

> Gareth,
>
> OK, there are a couple things here that I'm confused about. This is how I
> would understand things working.
>
> If you want to connect to any https endpoint, the agent will need SOME
> identity (so you have to give it SOME public/private key pair - which is
> what the Identity section does). It doesn't have to be the key-pair of the
> server (in fact, under normal situations it is not - the server is
> identified with its own public/private key and the client with another).
> But the point is, if you are connecting to an https endpoint, you can't
> leave Identity section out of the agent config.
>
> So when you say, "without the "Identity" configuration section set on the
> agent config, i'd get a TLS error" this is what I would expect. You can't
> leave the Identity section out when connecting via https because in that
> case the agent has no keys to talk TLS to the server.
>
> What does your agent config look like when you get things to work? (I
> assume you do get it to work because you said without the Identity you get
> a TLS error, which implies you do get it to work WITH an Identity section -
> is this correct?) What key files are you putting in the agent Identity when
> you get it to work?
>
> So I guess what I am saying is - have you tried to generate your own
> certificate and assigned it to your agent's Identity and then tried to
> connect to multiple https endpoints? Because as I mentioned earlier in
> another post, the agent today doesn't do server-cert verification, so it
> should "just work". You shouldn't need different Identities per endpoint.
> Once we add in verification, the endpoints you want to collect metrics from
> would need their server-side certs to be signed with a CA that the agent
> trusts (i.e. from the agent host's default root CA set) - we would then
> have to add the ability for the agent to be told about different CAs in
> case your server-side certs are, say, self-signed or signed with your own
> CA that isn't a trusted one found in the host's default root CA set.
>
> Oh, and, Merry Christmas!
>
> John Mazz
>
> ----- Original Message -----
> > One of the first services i am trying to monitor is etcd. etcd in OCP is
> > configured as per the below:
> >
> > /var/lib/origin/openshift.local.config/master/master-config.yaml
> >
> >
> > etcdClientInfo:
> >   ca: ca.crt
> >   certFile: master.etcd-client.crt
> >   keyFile: master.etcd-client.key
> >   urls:
> >   - https://10.2.2.2:4001
> >
> > Which responds with the below cURL:
> >
> > curl https://10.2.2.2:4001/metrics --cacert ./ca.crt --cert
> > ./master.etcd-client.crt --key ./master.etcd-client.key
> >
> > So without the "Identity" configuration section set on the agent config,
> > i'd get a TLS error. As etcd is a core part of OCP, I don't have much
> > control over the client certs and expect there might be other services
> > which require the same setup using different certs that i might want to
> > monitor.
> >
> > Hope that makes things clear, and Merry Christmas.
> >
> > Cheers.
> >
> > On Sat, Dec 24, 2016 at 3:30 PM, John Mazzitelli <mazz at redhat.com>
> wrote:
> >
> > > > Currently it seems you can only provide the agent configmap with the
> > > identity
> > > > field. But what i want to actually do, is provide this based on the
> pods
> > > > config map>
> > > > [chomp]
> > > > Is that possible? or planned for the future?
> > >
> > > I was hoping this wasn't going to be needed :) But we did talk about
> it.
> > >
> > > It is not possible today because there is one major problem with what
> you
> > > suggest that would need to be solved somehow:
> > >
> > > > cert_file: /var/run/secrets/client-crt/client.crt
> > > > private_key_file: /var/run/secrets/client-key/client.key
> > >
> > > That is inside your configmap on your OpenShift project (which may or
> may
> > > not be the same project where the agent is deployed).
> > >
> > > So - what file system is that actually referring to? And how does the
> > > agent get access to those files?
> > > _______________________________________________
> > > hawkular-dev mailing list
> > > hawkular-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20161228/73142866/attachment-0001.html