4:09 p.m.
We currently are currently doing both push (inventory) and pull (metrics).
Which means we are going to have to deal configuring things on both ends,
and handling security here might get interesting.
For push, we need to pass to the agent:
- the url for Hawkular Services
- the username & password
- the CA certificate (optional; if Hawkular Services is using tls with
untrusted certificates)
And we need to make sure that Hawkular Services is signed with a
certificate valid for its hostname and make sure its easy to export the CA
certificate so that its easy to pass on to the agents.
For pull, this might get a bit tricky.
To access a pod's metric endpoint we will need to do so using its ip
address, and to do this properly the certificate used for the metric
endpoint must be valid for that ip address. Since the ip address of a pod
is not known before a pod is created, this means we need something to
dynamically generate a certificate for us which we fetch at startup. This
also means we cannot have a common secret containing the certificate that
can be shared across replica sets.
To do this properly with pods may require a lot of extra effort. With
'pets' its a lot easier.
Even if we have properly signed certificates, there is also a question of
how we get the CA for those certificates into Prometheus.
Do we really need to have p8s trust the certificate for the endpoint which
is being exposed? Or could we configure p8s to trust any certificate
without validating it first? There is no extra verification if someone
decides to use a non-https endpoint for instance.
I see a few options here, but I might be missing other options as well:
1) by default we check for certificate validation, but we allow an override
to disable it. If someone really wants to use certificate validation with
pods, then they can figure out on their how to get the right certificates
into the pod to be used by the agent.
2) we provide some service which when an agent registers with inventory, we
generate a certificate and key they can use (signed by our own CA). The
metrics endpoint then uses this certificate.
3) we do something like not expose an http endpoint at the agent, but
tunnel this to Hawkular Services. P8s could then read the metric endpoints
directly from Hawkular Services.