6:45 a.m.
Currently it seems you can only provide the agent configmap with the
identity field. But what i want to actually do, is provide this based on
the pods config map, i.e.:
data:
hawkular-openshift-agent: |
endpoints:
- type: prometheus
protocol: "https"
port: 9779
path: /metrics
collection_interval_secs: 5
metrics:
- name: my-first-metric
type: counter
identity:
cert_file: /var/run/secrets/client-crt/client.crt
private_key_file: /var/run/secrets/client-key/client.key
The reason being, i might have multiple prometheus endpoints that have
different certs.
Is that possible? or planned for the future?
Cheers.
9:30 a.m.
Currently it seems you can only provide the agent configmap with the
identity
field. But what i want to actually do, is provide this based on the pods
config map>
[chomp]
Is that possible? or planned for the future?
I was hoping this wasn't going to be needed :) But we did talk about it.
It is not possible today because there is one major problem with what you suggest that
would need to be solved somehow:
cert_file: /var/run/secrets/client-crt/client.crt
private_key_file: /var/run/secrets/client-key/client.key
That is inside your configmap on your OpenShift project (which may or may not be the same
project where the agent is deployed).
So - what file system is that actually referring to? And how does the agent get access to
those files?
3:56 a.m.
One of the first services i am trying to monitor is etcd. etcd in OCP is
configured as per the below:
/var/lib/origin/openshift.local.config/master/master-config.yaml
etcdClientInfo:
ca: ca.crt
certFile: master.etcd-client.crt
keyFile: master.etcd-client.key
urls:
- https://10.2.2.2:4001
Which responds with the below cURL:
curl https://10.2.2.2:4001/metrics --cacert ./ca.crt --cert
./master.etcd-client.crt --key ./master.etcd-client.key
So without the "Identity" configuration section set on the agent config,
i'd get a TLS error. As etcd is a core part of OCP, I don't have much
control over the client certs and expect there might be other services
which require the same setup using different certs that i might want to
monitor.
Hope that makes things clear, and Merry Christmas.
Cheers.
On Sat, Dec 24, 2016 at 3:30 PM, John Mazzitelli <mazz(a)redhat.com> wrote:
> Currently it seems you can only provide the agent configmap with
the
identity
> field. But what i want to actually do, is provide this based on the pods
> config map>
> [chomp]
> Is that possible? or planned for the future?
I was hoping this wasn't going to be needed :) But we did talk about it.
It is not possible today because there is one major problem with what you
suggest that would need to be solved somehow:
> cert_file: /var/run/secrets/client-crt/client.crt
> private_key_file: /var/run/secrets/client-key/client.key
That is inside your configmap on your OpenShift project (which may or may
not be the same project where the agent is deployed).
So - what file system is that actually referring to? And how does the
agent get access to those files?
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
7:23 a.m.
Gareth,
OK, there are a couple things here that I'm confused about. This is how I would
understand things working.
If you want to connect to any https endpoint, the agent will need SOME identity (so you
have to give it SOME public/private key pair - which is what the Identity section does).
It doesn't have to be the key-pair of the server (in fact, under normal situations it
is not - the server is identified with its own public/private key and the client with
another). But the point is, if you are connecting to an https endpoint, you can't
leave Identity section out of the agent config.
So when you say, "without the "Identity" configuration section set on the
agent config, i'd get a TLS error" this is what I would expect. You can't
leave the Identity section out when connecting via https because in that case the agent
has no keys to talk TLS to the server.
What does your agent config look like when you get things to work? (I assume you do get it
to work because you said without the Identity you get a TLS error, which implies you do
get it to work WITH an Identity section - is this correct?) What key files are you putting
in the agent Identity when you get it to work?
So I guess what I am saying is - have you tried to generate your own certificate and
assigned it to your agent's Identity and then tried to connect to multiple https
endpoints? Because as I mentioned earlier in another post, the agent today doesn't do
server-cert verification, so it should "just work". You shouldn't need
different Identities per endpoint. Once we add in verification, the endpoints you want to
collect metrics from would need their server-side certs to be signed with a CA that the
agent trusts (i.e. from the agent host's default root CA set) - we would then have to
add the ability for the agent to be told about different CAs in case your server-side
certs are, say, self-signed or signed with your own CA that isn't a trusted one found
in the host's default root CA set.
Oh, and, Merry Christmas!
John Mazz
----- Original Message -----
One of the first services i am trying to monitor is etcd. etcd in OCP
is
configured as per the below:
/var/lib/origin/openshift.local.config/master/master-config.yaml
etcdClientInfo:
ca: ca.crt
certFile: master.etcd-client.crt
keyFile: master.etcd-client.key
urls:
- https://10.2.2.2:4001
Which responds with the below cURL:
curl https://10.2.2.2:4001/metrics --cacert ./ca.crt --cert
./master.etcd-client.crt --key ./master.etcd-client.key
So without the "Identity" configuration section set on the agent config,
i'd get a TLS error. As etcd is a core part of OCP, I don't have much
control over the client certs and expect there might be other services
which require the same setup using different certs that i might want to
monitor.
Hope that makes things clear, and Merry Christmas.
Cheers.
On Sat, Dec 24, 2016 at 3:30 PM, John Mazzitelli <mazz(a)redhat.com> wrote:
> > Currently it seems you can only provide the agent configmap with the
> identity
> > field. But what i want to actually do, is provide this based on the pods
> > config map>
> > [chomp]
> > Is that possible? or planned for the future?
>
> I was hoping this wasn't going to be needed :) But we did talk about it.
>
> It is not possible today because there is one major problem with what you
> suggest that would need to be solved somehow:
>
> > cert_file: /var/run/secrets/client-crt/client.crt
> > private_key_file: /var/run/secrets/client-key/client.key
>
> That is inside your configmap on your OpenShift project (which may or may
> not be the same project where the agent is deployed).
>
> So - what file system is that actually referring to? And how does the
> agent get access to those files?
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
5:02 a.m.
Hi John,
Below are the steps i am doing;
1. The certs for etcd are here (see path below), both server and client. I
am only interested in the client for the agent:
[root@localhost master]# pwd
/var/lib/origin/openshift.local.config/master
[root@localhost master]# ls -ltr *etc*
-rw-rw-rw-. 1 root root 1078 Sep 23 18:31 master.etcd-client.crt
-rw-rw-rw-. 1 root root 1679 Sep 23 18:31 master.etcd-client.key
-rw-rw-rw-. 1 root root 1675 Sep 23 18:31 etcd.server.key
-rw-rw-rw-. 1 root root 2550 Sep 23 18:31 etcd.server.crt
2. I add the client certs as secrets and mount them to the agent:
oc project openshift-infra
oc secrets new etcd-client-crt master.etcd-client.crt
oc secrets new etcd-client-key master.etcd-client.key
oc volume rc/hawkular-openshift-agent --add --name=etcd-client-crt
--type=secret --secret-name=etcd-client-crt
--mount-path=/run/secrets/etcd-client-crt
oc volume rc/hawkular-openshift-agent --add --name=etcd-client-key
--type=secret --secret-name=etcd-client-key
--mount-path=/run/secrets/etcd-client-key
3. Then edit the config map of the agent and add in the below, which
matches the above secret mounts:
oc edit configmap hawkular-openshift-agent-configuration
identity:
cert_file: /run/secrets/etcd-client-crt/master.etcd-client.crt
private_key_file: /run/secrets/etcd-client-key/master.etcd-client.key
4. Restart the pod to force a refresh and check the logs, which shows:
I1228 10:20:18.799687 1 prometheus_metrics_collector.go:97] DEBUG:
Told to collect [2] Prometheus metrics from [https://172.17.0.8:9779/metrics
]
I1228 10:20:18.984615 1 metrics_storage.go:152] DEBUG: Stored
datapoints for [2] metrics
I now have a working agent collecting from etcd.
Since etcd is mutual auth - maybe thats what is causing the confusion, as
you keep mentioning server-certs - i am not sure how generating my own
client certs helps.
But since you said try it, i have with the below commands but as i would
expect, got a TLS error:
openssl req -newkey rsa:2048 -nodes -keyout agent.key -out agent.csr -subj
"/C=UK/ST=Yorkshire/L=Leeds/O=Home/CN=hawkular-agent"
openssl x509 -signkey agent.key -in agent.csr -req -days 365 -out agent.crt
Just incase there are cross wires; etcd in OCP requires mutual auth (thats
how i understand it), so thats the reason i am adding in the etcd client
certs to the identity section of the agent configmap. If i needed to
monitor another endpoint which was also mutual auth, with the current setup
i wouldn't be able to do that.
If theres anything you want me to try, happy to do so.
Cheers.
On Sun, Dec 25, 2016 at 1:23 PM, John Mazzitelli <mazz(a)redhat.com> wrote:
Gareth,
OK, there are a couple things here that I'm confused about. This is how I
would understand things working.
If you want to connect to any https endpoint, the agent will need SOME
identity (so you have to give it SOME public/private key pair - which is
what the Identity section does). It doesn't have to be the key-pair of the
server (in fact, under normal situations it is not - the server is
identified with its own public/private key and the client with another).
But the point is, if you are connecting to an https endpoint, you can't
leave Identity section out of the agent config.
So when you say, "without the "Identity" configuration section set on the
agent config, i'd get a TLS error" this is what I would expect. You can't
leave the Identity section out when connecting via https because in that
case the agent has no keys to talk TLS to the server.
What does your agent config look like when you get things to work? (I
assume you do get it to work because you said without the Identity you get
a TLS error, which implies you do get it to work WITH an Identity section -
is this correct?) What key files are you putting in the agent Identity when
you get it to work?
So I guess what I am saying is - have you tried to generate your own
certificate and assigned it to your agent's Identity and then tried to
connect to multiple https endpoints? Because as I mentioned earlier in
another post, the agent today doesn't do server-cert verification, so it
should "just work". You shouldn't need different Identities per endpoint.
Once we add in verification, the endpoints you want to collect metrics from
would need their server-side certs to be signed with a CA that the agent
trusts (i.e. from the agent host's default root CA set) - we would then
have to add the ability for the agent to be told about different CAs in
case your server-side certs are, say, self-signed or signed with your own
CA that isn't a trusted one found in the host's default root CA set.
Oh, and, Merry Christmas!
John Mazz
----- Original Message -----
> One of the first services i am trying to monitor is etcd. etcd in OCP is
> configured as per the below:
>
> /var/lib/origin/openshift.local.config/master/master-config.yaml
>
>
> etcdClientInfo:
> ca: ca.crt
> certFile: master.etcd-client.crt
> keyFile: master.etcd-client.key
> urls:
> - https://10.2.2.2:4001
>
> Which responds with the below cURL:
>
> curl https://10.2.2.2:4001/metrics --cacert ./ca.crt --cert
> ./master.etcd-client.crt --key ./master.etcd-client.key
>
> So without the "Identity" configuration section set on the agent config,
> i'd get a TLS error. As etcd is a core part of OCP, I don't have much
> control over the client certs and expect there might be other services
> which require the same setup using different certs that i might want to
> monitor.
>
> Hope that makes things clear, and Merry Christmas.
>
> Cheers.
>
> On Sat, Dec 24, 2016 at 3:30 PM, John Mazzitelli <mazz(a)redhat.com>
wrote:
>
> > > Currently it seems you can only provide the agent configmap with the
> > identity
> > > field. But what i want to actually do, is provide this based on the
pods
> > > config map>
> > > [chomp]
> > > Is that possible? or planned for the future?
> >
> > I was hoping this wasn't going to be needed :) But we did talk about
it.
> >
> > It is not possible today because there is one major problem with what
you
> > suggest that would need to be solved somehow:
> >
> > > cert_file: /var/run/secrets/client-crt/client.crt
> > > private_key_file: /var/run/secrets/client-key/client.key
> >
> > That is inside your configmap on your OpenShift project (which may or
may
> > not be the same project where the agent is deployed).
> >
> > So - what file system is that actually referring to? And how does the
> > agent get access to those files?
> > _______________________________________________
> > hawkular-dev mailing list
> > hawkular-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> >
>
8:29 a.m.
OK, yes, its the mutual auth that I was missing. This is why you need to use that specific
client cert, because I assume the server-side will only trust that client cert and no
others (unless you somehow add your generated cert to some kind of "trust store"
so the server trusts it as well).
Hmm... so this is an interesting problem. For now, I would say the answer is
"generate a cert for your agent, and tell your servers to trust it" if that is
even possible but even so I assume that isn't a nice way to do it??
The problem with the agent is the endpoints are defined by your pods (they are config maps
on the projects where the pods are deployed) and the people doing those pod deployments
are typically not the same admin person who configures and sets up the agent. So how would
the pod configmap authors know the locations and names of the key-pairs or have privileges
to install those keys on the agent (I'm thinking of the use case where OpenShift is
running and developers/deployers are deploying their own applications and want to collect
their own app metrics).
Anyway, I created an issue in github where we can discuss this further there:
https://github.com/hawkular/hawkular-openshift-agent/issues/75
----- Original Message -----
Hi John,
Below are the steps i am doing;
1. The certs for etcd are here (see path below), both server and client. I
am only interested in the client for the agent:
[root@localhost master]# pwd
/var/lib/origin/openshift.local.config/master
[root@localhost master]# ls -ltr *etc*
-rw-rw-rw-. 1 root root 1078 Sep 23 18:31 master.etcd-client.crt
-rw-rw-rw-. 1 root root 1679 Sep 23 18:31 master.etcd-client.key
-rw-rw-rw-. 1 root root 1675 Sep 23 18:31 etcd.server.key
-rw-rw-rw-. 1 root root 2550 Sep 23 18:31 etcd.server.crt
2. I add the client certs as secrets and mount them to the agent:
oc project openshift-infra
oc secrets new etcd-client-crt master.etcd-client.crt
oc secrets new etcd-client-key master.etcd-client.key
oc volume rc/hawkular-openshift-agent --add --name=etcd-client-crt
--type=secret --secret-name=etcd-client-crt
--mount-path=/run/secrets/etcd-client-crt
oc volume rc/hawkular-openshift-agent --add --name=etcd-client-key
--type=secret --secret-name=etcd-client-key
--mount-path=/run/secrets/etcd-client-key
3. Then edit the config map of the agent and add in the below, which
matches the above secret mounts:
oc edit configmap hawkular-openshift-agent-configuration
identity:
cert_file: /run/secrets/etcd-client-crt/master.etcd-client.crt
private_key_file: /run/secrets/etcd-client-key/master.etcd-client.key
4. Restart the pod to force a refresh and check the logs, which shows:
I1228 10:20:18.799687 1 prometheus_metrics_collector.go:97] DEBUG:
Told to collect [2] Prometheus metrics from [https://172.17.0.8:9779/metrics
]
I1228 10:20:18.984615 1 metrics_storage.go:152] DEBUG: Stored
datapoints for [2] metrics
I now have a working agent collecting from etcd.
Since etcd is mutual auth - maybe thats what is causing the confusion, as
you keep mentioning server-certs - i am not sure how generating my own
client certs helps.
But since you said try it, i have with the below commands but as i would
expect, got a TLS error:
openssl req -newkey rsa:2048 -nodes -keyout agent.key -out agent.csr -subj
"/C=UK/ST=Yorkshire/L=Leeds/O=Home/CN=hawkular-agent"
openssl x509 -signkey agent.key -in agent.csr -req -days 365 -out agent.crt
Just incase there are cross wires; etcd in OCP requires mutual auth (thats
how i understand it), so thats the reason i am adding in the etcd client
certs to the identity section of the agent configmap. If i needed to
monitor another endpoint which was also mutual auth, with the current setup
i wouldn't be able to do that.
If theres anything you want me to try, happy to do so.
Cheers.
On Sun, Dec 25, 2016 at 1:23 PM, John Mazzitelli <mazz(a)redhat.com> wrote:
> Gareth,
>
> OK, there are a couple things here that I'm confused about. This is how I
> would understand things working.
>
> If you want to connect to any https endpoint, the agent will need SOME
> identity (so you have to give it SOME public/private key pair - which is
> what the Identity section does). It doesn't have to be the key-pair of the
> server (in fact, under normal situations it is not - the server is
> identified with its own public/private key and the client with another).
> But the point is, if you are connecting to an https endpoint, you can't
> leave Identity section out of the agent config.
>
> So when you say, "without the "Identity" configuration section set on
the
> agent config, i'd get a TLS error" this is what I would expect. You
can't
> leave the Identity section out when connecting via https because in that
> case the agent has no keys to talk TLS to the server.
>
> What does your agent config look like when you get things to work? (I
> assume you do get it to work because you said without the Identity you get
> a TLS error, which implies you do get it to work WITH an Identity section -
> is this correct?) What key files are you putting in the agent Identity when
> you get it to work?
>
> So I guess what I am saying is - have you tried to generate your own
> certificate and assigned it to your agent's Identity and then tried to
> connect to multiple https endpoints? Because as I mentioned earlier in
> another post, the agent today doesn't do server-cert verification, so it
> should "just work". You shouldn't need different Identities per
endpoint.
> Once we add in verification, the endpoints you want to collect metrics from
> would need their server-side certs to be signed with a CA that the agent
> trusts (i.e. from the agent host's default root CA set) - we would then
> have to add the ability for the agent to be told about different CAs in
> case your server-side certs are, say, self-signed or signed with your own
> CA that isn't a trusted one found in the host's default root CA set.
>
> Oh, and, Merry Christmas!
>
> John Mazz
>
> ----- Original Message -----
> > One of the first services i am trying to monitor is etcd. etcd in OCP is
> > configured as per the below:
> >
> > /var/lib/origin/openshift.local.config/master/master-config.yaml
> >
> >
> > etcdClientInfo:
> > ca: ca.crt
> > certFile: master.etcd-client.crt
> > keyFile: master.etcd-client.key
> > urls:
> > - https://10.2.2.2:4001
> >
> > Which responds with the below cURL:
> >
> > curl https://10.2.2.2:4001/metrics --cacert ./ca.crt --cert
> > ./master.etcd-client.crt --key ./master.etcd-client.key
> >
> > So without the "Identity" configuration section set on the agent
config,
> > i'd get a TLS error. As etcd is a core part of OCP, I don't have much
> > control over the client certs and expect there might be other services
> > which require the same setup using different certs that i might want to
> > monitor.
> >
> > Hope that makes things clear, and Merry Christmas.
> >
> > Cheers.
> >
> > On Sat, Dec 24, 2016 at 3:30 PM, John Mazzitelli <mazz(a)redhat.com>
> wrote:
> >
> > > > Currently it seems you can only provide the agent configmap with the
> > > identity
> > > > field. But what i want to actually do, is provide this based on the
> pods
> > > > config map>
> > > > [chomp]
> > > > Is that possible? or planned for the future?
> > >
> > > I was hoping this wasn't going to be needed :) But we did talk about
> it.
> > >
> > > It is not possible today because there is one major problem with what
> you
> > > suggest that would need to be solved somehow:
> > >
> > > > cert_file: /var/run/secrets/client-crt/client.crt
> > > > private_key_file: /var/run/secrets/client-key/client.key
> > >
> > > That is inside your configmap on your OpenShift project (which may or
> may
> > > not be the same project where the agent is deployed).
> > >
> > > So - what file system is that actually referring to? And how does the
> > > agent get access to those files?
> > > _______________________________________________
> > > hawkular-dev mailing list
> > > hawkular-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > >
> >
>
1:32 p.m.
BTW: I would like to know more about why you want this.
The "Identity" configuration identifies the agent (so having one key-pair makes
sense - it identifies your agent. Having multiple key-pairs per agent will actually mean
your agent has different identities depending on what endpoint it is talking to - not sure
this is what we want).
If you have multiple Prometheus endpoints (each with their own server key/cert) I
don't see why you would need different agent identities defined in your endpoints. The
"identity" is the client's identification, nothing to do with the server,
and a client should have one identity, not multiple.
Now, if the concern is that your different Prometheus endpoint server certs are signed by
different CAs (or are all self-signed) that is a different issue I think. It is assumed
the host's default root CA set would be good enough to verify server endpoints, but if
not, we would need to provide to the agent with all the CA certificates necessary for
endpoints to be verified. Note: for the record, the agent doesn't do any server
verification today - see
https://github.com/hawkular/hawkular-openshift-agent/blob/master/http/htt... -
so the agent should be able to collect metrics from any endpoint today.
In the future we would need to be able to provide the agent with a trust store that
contains all the CA certs required to talk to all the endpoints, assuming the host's
default root CA set is not good enough. This is what we haven't implemented yet.
Probably something like "ca_cert_file" defined in the "Identity"
section, which would mean the Identity section would not only tell the agent what its own
key-pair is, but will also say what its trusted CAs are.
----- Original Message -----
Currently it seems you can only provide the agent configmap with the
identity
field. But what i want to actually do, is provide this based on the pods
config map, i.e.:
data:
hawkular-openshift-agent: |
endpoints:
- type: prometheus
protocol: "https"
port: 9779
path: /metrics
collection_interval_secs: 5
metrics:
- name: my-first-metric
type: counter
identity:
cert_file: /var/run/secrets/client-crt/client.crt
private_key_file: /var/run/secrets/client-key/client.key
The reason being, i might have multiple prometheus endpoints that have
different certs.
Is that possible? or planned for the future?
Cheers.
2890
days inactive
2894
days old
6 comments
2 participants
participants (2)
-
Gareth Healy -
John Mazzitelli