## Summary Attempts to run gitlab-rake gitlab:backup:create fails with: Backup::Error: Backup operation failed: gtar: /srv/gitlab/public/uploads: Cannot open: Permission denied This error occurs with the 13.11.0 task runner container. I have not tested this against other container versions. ## Steps to reproduce 1. Deploy with Helm, 13.11.0 2. Get the task-runner pod 3. Execute a backup (kubectl exec -it task-runner-pod-name -- gitlab-rake gitlab:backup:create ## What is the current bug behavior? ``` 2021-06-18 13:43:09 +0000 -- Dumping uploads ... rake aborted! Backup::Error: Backup operation failed: gtar: /srv/gitlab/public/uploads: Cannot open: Permission denied gtar: Error is not recoverable: exiting now /srv/gitlab/lib/backup/files.rb:52:in `dump' /srv/gitlab/lib/tasks/gitlab/backup.rake:186:in `block (4 levels) in <main>' /srv/gitlab/lib/tasks/gitlab/backup.rake:14:in `block (3 levels) in <main>' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>' /srv/gitlab/bin/bundle:3:in `load' /srv/gitlab/bin/bundle:3:in `<main>' Tasks: TOP => gitlab:backup:uploads:create (See full trace by running task with --trace) command terminated with exit code 1 ``` In tracing this versus the commercial containers I use in my lab, the /srv/gitlab/public/uploads directory is owned by the git user, with user-only access (700) in the non-hardened containers, and by root in the Ironbank hardened containers, with root-only access (700). The backup task runs as ## What is the expected correct behavior? ``` 2021-06-17 20:45:36 +0000 -- Dumping uploads ... <probably some stuff here if you've got uploads> 2021-06-17 20:45:36 +0000 -- done ``` ## Relevant logs and/or screenshots Running it with --trace output provides: ``` ** Invoke gitlab:backup:uploads:create (first_time) ** Invoke gitlab_environment ** Execute gitlab:backup:uploads:create 2021-06-18 13:49:13 +0000 -- Dumping uploads ... rake aborted! Backup::Error: Backup operation failed: gtar: /srv/gitlab/public/uploads: Cannot open: Permission denied gtar: Error is not recoverable: exiting now /srv/gitlab/lib/backup/files.rb:52:in `dump' /srv/gitlab/lib/tasks/gitlab/backup.rake:186:in `block (4 levels) in <main>' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `block in execute' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `each' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `execute' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke' /srv/gitlab/lib/tasks/gitlab/backup.rake:14:in `block (3 levels) in <main>' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `block in execute' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `each' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `execute' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `each' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:83:in `block in run' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:80:in `run' /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>' /srv/gitlab/vendor/bundle/ruby/2.7.0/bin/rake:23:in `load' /srv/gitlab/vendor/bundle/ruby/2.7.0/bin/rake:23:in `<top (required)>' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors' /usr/lib64/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>' /srv/gitlab/bin/bundle:3:in `load' /srv/gitlab/bin/bundle:3:in `<main>' Tasks: TOP => gitlab:backup:uploads:create command terminated with exit code 1 ``` ## Possible fixes Provide non-root read access to the uploads directory (probably need to provide write access to or the restore will likely fail). Not sure if this is also true for artifacts, etc. ## Defintion of Done - [ ] Bug has been identified and corrected within the container ## Other Notes I do realize that the backup-utility exists and is what we should use, however, since it doesn't support Azure Blob Storage that we're using to back our instance (versus MinIO or AWS S3), I'm trying to find other ways to backup. /cc @ironbank-notifications/bug