failed deployment due to invalid pod labels

Bug

Description

Describe the problem, what were you doing when you noticed the bug?

Upgrading from bigbang 2.34.0 to 2.35.0 with kiali enabled caused helm failure and rollback of velero

Provide any steps possible used to reproduce the error (ideally in an isolated fashion).

Upgrade from BB 2.34.0 to 2.35.0 with kiali enabled

The specific error

Upgrade "velero-velero" failed: cannot patch "node-agent" with kind DaemonSet: DaemonSet.apps "node-agent" is invalid: [spec.template.labels: Invalid value: "{{ .Chart.Name }}": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?'), spec.template.labels: Invalid value: "{{ .Chart.AppVersion }}": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]

Introduced recently, Kiali defines kialiAppLabel & kialiVersionLabel as {{ .Chart.Name }} and {{ .Chart.AppVersion }} respectively

https://repo1.dso.mil/big-bang/bigbang/-/blob/master/chart/templates/_helpers.tpl?ref_type=heads#L233-253

{{/*
App Label for Kiali trace correlation
To be used for Kiali-required labels on pods
This will:
  * enable proper linking of Jaeger traces in Kiali
  * enable full Kiali label tracking of pods
*/}}
{{- define "kialiAppLabel" -}}
app: {{ "{{ .Chart.Name }}" | quote }}
{{- end -}}

{{/*
Version label for Kiali trace correlation
To be used for Kiali-required labels on pods
This will:
  * enable proper linking of Jaeger traces in Kiali
  * enable full Kiali label tracking of pods
*/}}
{{- define "kialiVersionLabel" -}}
version: {{ "{{ .Chart.AppVersion }}" | quote }}
{{- end -}}

If Kiali is enabled, velero set pod labels based on these values

https://repo1.dso.mil/big-bang/bigbang/-/blob/master/chart/templates/velero/values.yaml?ref_type=heads#L22-26

{{- if .Values.kiali.enabled }}
podLabels:
  {{- include "kialiAppLabel" . | nindent 2 }}
  {{- include "kialiVersionLabel" . | nindent 2 }}
{{- end }}

Velero then inserts the values {{ .Chart.Name }} and {{ .Chart.AppVersion }} as literal values rather than evaluating them.

https://repo1.dso.mil/big-bang/product/packages/velero/-/blob/main/chart/templates/node-agent-daemonset.yaml?ref_type=heads#L33-35

      {{- if .Values.podLabels }}
        {{- toYaml .Values.podLabels | nindent 8 }}
      {{- end }}

Other Charts (eg Keycloak and Fluentbit) evaluate Go template snippets in .Values.podLabels (using tpl function)

https://repo1.dso.mil/big-bang/product/packages/keycloak/-/blob/main/chart/templates/statefulset.yaml?ref_type=heads#L41-43

        {{- range $key, $value := .Values.podLabels }}
        {{- printf "%s: %s" $key (tpl $value $ | quote) | nindent 8 }}
        {{- end }}

https://repo1.dso.mil/big-bang/product/packages/fluentbit/-/blob/main/chart/templates/daemonset.yaml?ref_type=heads#L31-33

        {{- with .Values.podLabels }}
        {{- tpl (toYaml .) $ | nindent 8 }}
        {{- end }}

Temporary workaround, add this to the values file

velero:
  values:
    podLabels:
      app: velero
      version: 1.14.1

BigBang Version

What version of BigBang were you running?

This can be retrieved multiple ways:

# via helm
helm ls -n bigbang

# via the deployed umbrella git tag
kubectl get gitrepository -n bigbang