TL;DR: We forked the grafana build tools [0] and built a commit off the master branch that we knew to be stable. In addition we modified the build scripts to include some Grafana plugins inside the container image. There were two reasons we did this.
1. We were depending on new features in Grafana that were only available on the master branch. Grafana was under very heavy development as they geared up towards their 5.0 release and we were finding new problems arising every few days while we worked off the master branch. This was really slowing down APB development.
2. We needed to install additional plugins required by some dashboards.
Grafana's docker image provides a way to install plugins using environment variables fed into the container at runtime but this requires root access which is not feasible in Openshift. The only way we could do it at the time was by baking in the plugins as part of the docker build. It's a terrible solution but it was the only way at the time to continue the very fast pace of development of the APBs.
There are still a some questions to be answered:
* How are we going to distribute Grafana on OpenShift such that it doesn't require root access?
* What is the best way to install plugins? There was some discussion about the different options and the consequences of those options in this document [1]
This all arose during a sprint where one of the primary goals was to deliver an APB that could provision Grafana + Prometheus and display dashboard discovery and some other features. We decided at the time that we were not prepared to answer those bigger questions so we knowingly put this technical debt on ourselves and implemented a quick and dirty solution.