Migrations Job Istio Sidecar exits prematurely
Summary
The changes made to improve the interaction between the migration
container and the istio-proxy
container that are started as part of the istio-injected migrations
Job are resulting in the following error during the execution of the migrations
Job, which causes the Job (and therefore the flux chart upgrade) to fail:
> k logs -n gitlab gitlab-migrations-7-z72gn -f
Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Attempting to run '/scripts/wait-for-deps /scripts/db-migrate && sleep 5 && echo "Attempting to stop the istio proxy..." && echo "curl -X POST http://localhost:15020/quitquitquit" && curl -X POST http://localhost:15020/quitquitquit' as a main process
Checking: resque.yml, cable.yml
- FAILED connecting to 'redis://gitlab-redis-master.gitlab.svc:6379' from resque.yml, through gitlab-redis-master.gitlab.svc
ERROR: Error connecting to Redis on gitlab-redis-master.gitlab.svc:6379 (Errno::ECONNREFUSED)
- FAILED connecting to 'redis://gitlab-redis-master.gitlab.svc:6379' from cable.yml, through gitlab-redis-master.gitlab.svc
ERROR: Error connecting to Redis on gitlab-redis-master.gitlab.svc:6379 (Errno::ECONNREFUSED)
Checking: resque.yml, cable.yml
- FAILED connecting to 'redis://gitlab-redis-master.gitlab.svc:6379' from resque.yml, through gitlab-redis-master.gitlab.svc
ERROR: Error connecting to Redis on gitlab-redis-master.gitlab.svc:6379 (Errno::ECONNREFUSED)
- FAILED connecting to 'redis://gitlab-redis-master.gitlab.svc:6379' from cable.yml, through gitlab-redis-master.gitlab.svc
ERROR: Error connecting to Redis on gitlab-redis-master.gitlab.svc:6379 (Errno::ECONNREFUSED)
Checking: main
Checking: resque.yml, cable.yml
Looking at the upstream script used for the migrations job, it takes other scripts as arguments and runs them in the background. Because the execution of these jobs is not run in the foreground process, the commands used to shut down the istio sidecar upon job the completion of these scripts is called before they complete, which kills the jobs ability to reach anything and results in the above error.
Suggested Fix
Updating the arguments for the effected jobs to include a /bin/bash -c
command as follows:
args:
- /bin/bash
- -c
- /scripts/wait-for-deps
{{- if include "gitlab.geo.secondary" $ }}
- /scripts/geo-db-migrate
{{- else }}
- /scripts/db-migrate
{{- end }}
{{- if and .Values.global.istio.enabled (eq .Values.global.istio.injection "enabled") }}
- '&& sleep 5'
- '&& echo "Attempting to stop the istio proxy..."'
- '&& echo "curl -X POST http://localhost:15020/quitquitquit"'
- '&& curl -X POST http://localhost:15020/quitquitquit'
{{ - end }}