UNCLASSIFIED - NO CUI

Velero Container Permission & Functionality Issues

Summary

Executing Velero backups with the use of Kopia within the Velero container fails when attempting to write to a Minio S3 backend. Two issues were uncovered:

  1. Both the default 65534 and 0 users are unable to write to the root directory and create the udmrepo folder.
  2. Even after manually chmod 777 the root directory within the Velero container we are able to create the udmrepo directory but the kopia backups hang indefinitely and never complete.

The above issues are not present with the default Tanzu helmchart/containers

Steps to reproduce

  • Minio Instance available for S3 backend.
  • Velero CLI available to interact with Velero, or can use the Velero container itself.

Sample Helm Values:

`velero: image: repository: hidden/ext.registry1.dso.mil/ironbank/opensource/velero/velero tag: v1.17.0

podSecurityContext: runAsUser: 0 runAsGroup: 0

containerSecurityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL readOnlyRootFilesystem: false

configuration: credentials: useSecret: true existingSecret: velero-bucket-secret

backupStorageLocation:
  - name: default
    provider: aws
    bucket: dev-k8s-platform
    default: true
    accessMode: ReadWrite
    credential:
      name: velero-bucket-secret
      key: credentials-velero
    config:
      region: minio
      s3ForcePathStyle: "true"
      s3Url: http://hidden:9000
    annotations: {}
   
volumeSnapshotLocation: []

rbac: clusterAdministrator: true clusterAdministratorName: k8s_admin

credentials: useSecret: false

deployNodeAgent: true

nodeAgent: podSecurityContext: runAsUser: 0 fsGroup: 1337 containerSecurityContext: privileged: true

initContainers: - name: velero-plugin-for-aws image: hidden/ext.registry1.dso.mil/velero/velero-plugin-for-aws:v1.13.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins kubectl: image: repository: hidden/ext.hub.docker.com/bitnamilegacy/kubectl containerSecurityContext: capabilities: drop: - ALL resources: requests: memory: 256Mi cpu: 100m limits: memory: 256Mi cpu: 100m

upgradeCRDs: false `

  • Backup command:

velero backup create test1 --include-namespaces=monitoring --default-volumes-to-fs-backup --wait

What is the current bug behavior?

Within the Velero container the following message is thrown and on the CLI the backup fails

time="2025-10-09T09:20:00Z" level=info msg="Summary for skipped PVs: [{\"name\":\"pvc-e3ed4904-634d-4586-8a73-4035d56b9d90\",\"reasons\":[{\"approach\":\"podvolume\",\"reason\":\"encountered a problem with backing up the PVC of pod monitoring/kube-prometheus-stack-grafana-58c8c6c5c5-2gnr7: failed to wait BackupRepository, errored early: backup repository is not ready: error to connect to backup repo: error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /root/udmrepo: permission denied\"},{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]}]" backup=velero/test1 logSource="pkg/backup/backup.go:614" time="2025-10-09T09:20:00Z" level=info msg="Backed up a total of 248 items" backup=velero/test1 logSource="pkg/backup/backup.go:618" progress=

What is the expected correct behavior?

The expectation is that the backup would complete and when the backup is kicked off it can create the udmrepo directory within the Velero container,

After Reverting back to the Tanzu containers

NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR test1 PartiallyFailed 5 33 2025-10-09 05:19:57 -0400 EDT 29d default test3 Completed 0 0 2025-10-09 06:06:52 -0400 EDT 29d default test4 Completed 0 0 2025-10-09 06:10:00 -0400 EDT 29d default

Relevant logs and/or screenshots

sample CLI output of backup hanging after manually updating the /root directory with chmod 777. velero backup create test6 --include-namespaces=monitoring --default-volumes-to-fs-backup --wait Backup request "test6" submitted successfully. Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background. ................................................................................................................................................................................ ................................................................................................................................................................................ ................................................................................................................................................................................ ................................................................................................................................................................................ ................................................................................................................................................................................ ................

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Tasks

  • Bug has been identified and corrected within the container

Please read the Iron Bank Documentation for more info

UNCLASSIFIED - NO CUI