grafana persistence should use ironbank image for init

Enabling persistence for Grafana adds an init container which attempts to use busybox:1.31.1 (not an Ironbank image). This conflicts with the allowedDockerRegistries configured in Gatekeeper by default. I recommend we update the Grafana chart values to use an Ironbank approved image.

Relevant links:

• grafana chart template
• values

Bigbang values to enable Grafana persistence:

monitoring:
  values:
    grafana:
      persistence:
        enabled: true

Enabling persistence resulted in the following error:

7m26s       Warning   FailedCreate           replicaset/monitoring-monitoring-grafana-58c9d65468              Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [allowed-docker-registries] container <init-chown-data> has an invalid repository for  image  <busybox:1.31.1>, allowed repos are ["registry1.dso.mil", "registry.dso.mil"]

Values which resolved the issue:

monitoring:
  values:
    grafana:
      persistence:
        enabled: true
      initChownData:
        image:
          repository: registry1.dso.mil/ironbank/redhat/ubi/ubi8-minimal
          tag: "8.5"

There may be a more appropriate image to use in place of ubi8-minimal - I just grabbed the first one I thought would work.

added gatekeeper monitoring labels

That image just executes a chown on the PVC?

That appears to be all.. yea.

I'm open to other solutions here - changing the image to IB approved just seemed like the quickest fix.

You might also be able to turn off this init job so it doesn't run the chown each time:

https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L314

I could definitely disable it, but:

1. I'm not sure what impact, if any, that would have on functionality (if the chown isn't required, the init container should just be removed entirely IMO)
2. It seems to be undesirable for a BB core chart to have a feature that tries to use a non-IB image

To clarify - I understand the Grafana chart itself is out of our direct control..

Pending an update to the upstream (which seems unlikely) I think it might make sense to add the override as a default in the monitoring chart's Grafana values.

Or perhaps conditionally in the BB chart's monitoring values.

I think it would be best to disable the init/chown container even if that means adding in some values and conditionals above upstream, this isn't the first implementation or direction the upstream monitoring/grafana/etc charts have taken that we've had to deviate from, if the securityContext is set correctly (and not then altered) on a new install of Grafana then it will be fine. initContainer should only be needed when user was set differently/not-at-all/or-persistence-disabled-then-enabled.

Also I could have sworn we already had an issue for Grafana persistence but I can't find issues or recs anywhere.

added kindbug priority6 teamXForce labels

added BB Customer Issues label

removed kindbug label

@ablanchard this issue has been inactive for 30 days and is being labelled as stale. If this issue is still required please take action by removing the stale label and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken it will be auto closed in 60 days.

added stale label

set weight to 3

changed iteration to Big Bang Iterations Sep 6, 2022 - Sep 19, 2022

changed milestone to %1.43.0

We will need to test and document outcomes of disabling this initContainer (even if out of band) And testing an upgrade post changes, and from an install from main branch. I believe that since we set securityContext for each portion of the pods and containers we shouldn't need a chown but we'll need to confirm against a current main install and upgrade to using persistence without the init job present.

removed stale label

assigned to @ryan.j.garcia

added statusdoing label