For push, we need to pass to the agent:

- the url for Hawkular Services

- the username & password

- the CA certificate (optional; if Hawkular Services is using tls with untrusted certificates)

And we need to make sure that Hawkular Services is signed with a certificate valid for its hostname and make sure its easy to export the CA certificate so that its easy to pass on to the agents.

For pull, this might get a bit tricky.

To access a pod's metric endpoint we will need to do so using its ip address, and to do this properly the certificate used for the metric endpoint must be valid for that ip address. Since the ip address of a pod is not known before a pod is created, this means we need something to dynamically generate a certificate for us which we fetch at startup. This also means we cannot have a common secret containing the certificate that can be shared across replica sets.

To do this properly with pods may require a lot of extra effort. With 'pets' its a lot easier.

Even if we have properly signed certificates, there is also a question of how we get the CA for those certificates into Prometheus. 

Do we really need to have p8s trust the certificate for the endpoint which is being exposed? Or could we configure p8s to trust any certificate without validating it first? There is no extra verification if someone decides to use a non-https endpoint for instance.

I see a few options here, but I might be missing other options as well:

1) by default we check for certificate validation, but we allow an override to disable it. If someone really wants to use certificate validation with pods, then they can figure out on their how to get the right certificates into the pod to be used by the agent.

2) we provide some service which when an agent registers with inventory, we generate a certificate and key they can use (signed by our own CA). The metrics endpoint then uses this certificate.

3) we do something like not expose an http endpoint at the agent, but tunnel this to Hawkular Services. P8s could then read the metric endpoints directly from Hawkular Services.