[Hawkular-dev] Metric endpoint access

For Hawkular Services, we want to be able to handle monitoring EAP instances no matter where they are running. So we could have some eap instances running on bare metal, running in a vm, running as docker images somewhere, running in various OpenShift or kubernetes clusters. For baremetal and vm instances, this should be similar to how we have handled them in the past. For OpenShift or Kubernetes, I am not sure if we have figure out how this should function. Particularly with metric endpoints that need to be accessed from outside of the OpenShift cluster. If we are running Hawkular Services in an OpenShift cluster and monitoring eap pods within that cluster, by default Hawkular Services should be able to communicate with all the eap pods in the cluster by their ip address. So this is not much of an issue. But, if the ovs-multitenant SDN plugin is enabled instead, then only pods within the same project can communicate with each other. So if we are running Hawkular Services in one project we cannot reach the metric endpoint of eap instances running in another project. Running Hawkular Services in the 'default' project (vnid0) gives it special privileges to read from any pod, but this also means that only admins will be able to install this. There is also the new ovs-networkpolicy plugin, which allows for Kubernetes network policy. And this may further limit communication between pods. If we move Hawkular Services outside of the OpenShift cluster, then this can get tricky and I don't know what we can really do here. Even if we were to have Hawkular Services run with the same network setup as OpenShift (so it can access the pod endpoints) I don't think we can do this with multiple OpenShift instances. Normally, if you want to expose something outside of an OpenShift cluster, you would do so using a route. But this is not going to work for individual pods in a replica set. There is also the API proxy that could be used to access individual pod endpoints, but I think this could cause a performance problem. And the agent may not know the endpoint to tell p8s to start scraping from. Has anyone started to look into this yet?