<div dir="ltr">Hi John,<div><br></div><div>Below are the steps i am doing;</div><div><br></div><div>1. The certs for etcd are here (see path below), both server and client. I am only interested in the client for the agent:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>[root@localhost master]# pwd</div></div><div><div>/var/lib/origin/openshift.local.config/master</div></div><div><div><br></div></div><div><div>[root@localhost master]# ls -ltr *etc*</div></div><div><div>-rw-rw-rw-. 1 root root 1078 Sep 23 18:31 master.etcd-client.crt</div></div><div><div>-rw-rw-rw-. 1 root root 1679 Sep 23 18:31 master.etcd-client.key</div></div><div><div>-rw-rw-rw-. 1 root root 1675 Sep 23 18:31 etcd.server.key</div></div><div><div>-rw-rw-rw-. 1 root root 2550 Sep 23 18:31 etcd.server.crt</div></div><div><br></div></blockquote>2. I add the client certs as secrets and mount them to the agent:<div><br><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>oc project openshift-infra</div><div>oc secrets new etcd-client-crt master.etcd-client.crt</div><div>oc secrets new etcd-client-key master.etcd-client.key</div><div><br></div><div>oc volume rc/hawkular-openshift-agent --add --name=etcd-client-crt --type=secret --secret-name=etcd-client-crt --mount-path=/run/secrets/etcd-client-crt</div><div>oc volume rc/hawkular-openshift-agent --add --name=etcd-client-key --type=secret --secret-name=etcd-client-key --mount-path=/run/secrets/etcd-client-key</div><div><br></div></blockquote>3. Then edit the config map of the agent and add in the below, which matches the above secret mounts:<br><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div>oc edit configmap hawkular-openshift-agent-configuration<br></div><div><br></div><div>identity:</div><div> cert_file: /run/secrets/etcd-client-crt/master.etcd-client.crt</div><div> private_key_file: /run/secrets/etcd-client-key/master.etcd-client.key</div><div><br></div></blockquote>4. Restart the pod to force a refresh and check the logs, which shows:<br><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>I1228 10:20:18.799687 1 prometheus_metrics_collector.go:97] DEBUG: Told to collect [2] Prometheus metrics from [<a href="https://172.17.0.8:9779/metrics">https://172.17.0.8:9779/metrics</a>]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>I1228 10:20:18.984615 1 metrics_storage.go:152] DEBUG: Stored datapoints for [2] metrics</div><div><br></div></blockquote>I now have a working agent collecting from etcd.</div><div><br></div><div>Since etcd is mutual auth - maybe thats what is causing the confusion, as you keep mentioning server-certs - i am not sure how generating my own client certs helps.</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><br></blockquote><div>But since you said try it, i have with the below commands but as i would expect, got a TLS error:</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">openssl req -newkey rsa:2048 -nodes -keyout agent.key -out agent.csr -subj "/C=UK/ST=Yorkshire/L=Leeds/O=Home/CN=hawkular-agent"<div>openssl x509 -signkey agent.key -in agent.csr -req -days 365 -out agent.crt</div><div><br></div></blockquote><div>Just incase there are cross wires; etcd in OCP requires mutual auth (thats how i understand it), so thats the reason i am adding in the etcd client certs to the identity section of the agent configmap. If i needed to monitor another endpoint which was also mutual auth, with the current setup i wouldn't be able to do that.</div><div><br></div><div>If theres anything you want me to try, happy to do so.</div><div><br></div><div>Cheers.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 25, 2016 at 1:23 PM, John Mazzitelli <span dir="ltr"><<a href="mailto:mazz@redhat.com" target="_blank">mazz@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Gareth,<br>
<br>
OK, there are a couple things here that I'm confused about. This is how I would understand things working.<br>
<br>
If you want to connect to any https endpoint, the agent will need SOME identity (so you have to give it SOME public/private key pair - which is what the Identity section does). It doesn't have to be the key-pair of the server (in fact, under normal situations it is not - the server is identified with its own public/private key and the client with another). But the point is, if you are connecting to an https endpoint, you can't leave Identity section out of the agent config.<br>
<br>
So when you say, "without the "Identity" configuration section set on the agent config, i'd get a TLS error" this is what I would expect. You can't leave the Identity section out when connecting via https because in that case the agent has no keys to talk TLS to the server.<br>
<br>
What does your agent config look like when you get things to work? (I assume you do get it to work because you said without the Identity you get a TLS error, which implies you do get it to work WITH an Identity section - is this correct?) What key files are you putting in the agent Identity when you get it to work?<br>
<br>
So I guess what I am saying is - have you tried to generate your own certificate and assigned it to your agent's Identity and then tried to connect to multiple https endpoints? Because as I mentioned earlier in another post, the agent today doesn't do server-cert verification, so it should "just work". You shouldn't need different Identities per endpoint. Once we add in verification, the endpoints you want to collect metrics from would need their server-side certs to be signed with a CA that the agent trusts (i.e. from the agent host's default root CA set) - we would then have to add the ability for the agent to be told about different CAs in case your server-side certs are, say, self-signed or signed with your own CA that isn't a trusted one found in the host's default root CA set.<br>
<br>
Oh, and, Merry Christmas!<br>
<br>
John Mazz<br>
<div class="HOEnZb"><div class="h5"><br>
----- Original Message -----<br>
> One of the first services i am trying to monitor is etcd. etcd in OCP is<br>
> configured as per the below:<br>
><br>
> /var/lib/origin/openshift.<wbr>local.config/master/master-<wbr>config.yaml<br>
><br>
><br>
> etcdClientInfo:<br>
> ca: ca.crt<br>
> certFile: master.etcd-client.crt<br>
> keyFile: master.etcd-client.key<br>
> urls:<br>
> - <a href="https://10.2.2.2:4001" rel="noreferrer" target="_blank">https://10.2.2.2:4001</a><br>
><br>
> Which responds with the below cURL:<br>
><br>
> curl <a href="https://10.2.2.2:4001/metrics" rel="noreferrer" target="_blank">https://10.2.2.2:4001/metrics</a> --cacert ./ca.crt --cert<br>
> ./master.etcd-client.crt --key ./master.etcd-client.key<br>
><br>
> So without the "Identity" configuration section set on the agent config,<br>
> i'd get a TLS error. As etcd is a core part of OCP, I don't have much<br>
> control over the client certs and expect there might be other services<br>
> which require the same setup using different certs that i might want to<br>
> monitor.<br>
><br>
> Hope that makes things clear, and Merry Christmas.<br>
><br>
> Cheers.<br>
><br>
> On Sat, Dec 24, 2016 at 3:30 PM, John Mazzitelli <<a href="mailto:mazz@redhat.com">mazz@redhat.com</a>> wrote:<br>
><br>
> > > Currently it seems you can only provide the agent configmap with the<br>
> > identity<br>
> > > field. But what i want to actually do, is provide this based on the pods<br>
> > > config map><br>
> > > [chomp]<br>
> > > Is that possible? or planned for the future?<br>
> ><br>
> > I was hoping this wasn't going to be needed :) But we did talk about it.<br>
> ><br>
> > It is not possible today because there is one major problem with what you<br>
> > suggest that would need to be solved somehow:<br>
> ><br>
> > > cert_file: /var/run/secrets/client-crt/<wbr>client.crt<br>
> > > private_key_file: /var/run/secrets/client-key/<wbr>client.key<br>
> ><br>
> > That is inside your configmap on your OpenShift project (which may or may<br>
> > not be the same project where the agent is deployed).<br>
> ><br>
> > So - what file system is that actually referring to? And how does the<br>
> > agent get access to those files?<br>
> > ______________________________<wbr>_________________<br>
> > hawkular-dev mailing list<br>
> > <a href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
> > <a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>