Steven Tobin and I had a conversation about how to approach this task. We've decided to investigate the approach of using a sidecar service that runs inside the same pod as Grafana.

The idea is this service would use the k8's API to watch the current namespace for Config Maps that contain a Grafana dashboard definition (JSON), identifiable by an annotation. For example:

org.aerogear.grafana.dashboard: true

The sidecar service would scan the namespace for these Config Maps and feed the dashboard JSON into Grafana via an API call.

The beauty to this approach is that the dashboard JSON would be bundled either with the service or with the service APB. This reduces the potential for version mismatches between a service its associated Grafana dashboard. It also means downstream users could define their own dashboards for their services and have them auto-loaded into Grafana.

The downside is we'll potentially need to build our own service (or flesh out an existing one) for this approach to work. To validate the idea we are going to try this existing service: https://github.com/PierreVincent/k8s-grafana-watcher
This service is written in Golang and is a very basic implementation. For example, It does not handle updates or deletion of Config Maps.

Here's what we are going to do:

1. Create a forked version of the metrics-apb that also provisions the sidecar service linked above.
2 . Use another apb to create a config map containing an annotation and a sample dashboard JSON
3. Demonstrate auto loading of the dashboard into Grafana
4. Record a very short demo of this in action
5. Document any insights/gotchas we found along the way.

David Martin how does that sound to you?