nginx-alpine no longer start up

Summary

We have three images based on the nginx-alpine image (this one - dai-ai-ui, dai-ui and dai-docs. Up until a week ago, they were all working fine. However, starting on the 8th of June, the images produced are no longer usable. They still build ok, and the pipeline passes, but the published images are no longer functional.

Other things worth noting:

• I can build images from the same docker files locally and they run without issue.
• We haven't made any changes to our repos since it was working.

As I can't reproduce issue on locally built images I'm a bit stuck as how to diagnose the issue. Are there any infrastructure changes on the Iron Bank side that could explain this (between Friday 7th June and Saturday 8th of June)?

Link to failed pipeline

https://repo1.dso.mil/dsop/keysight/eggplant/dai-ai-ui/-/pipelines/3374273

Note pipeline is passing - but failing in the sense it stopped producing usable images.

What is the current bug behavior?

Running an image based on the latest build fails immediately:

e.g. pulling and running the latest image, it exits immediately.

docker run registry1.dso.mil/ironbank/keysight/eggplant/dai-ai-ui:7.3.0-3
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: ERROR: /etc/nginx/templates exists, but /etc/nginx/conf.d is not writable
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

What is the expected correct behavior?

Running an image built last week (7th Jubne) starts up fine as expected:

docker run -it  registry1.dso.mil/ironbank/keysight/eggplant/dai-ai-ui@sha256:f94019e64fd6e1686f998893f9e57a1cdf079732228bfc1ade873ee059c702be
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: ERROR: /etc/nginx/templates exists, but /etc/nginx/conf.d is not writable
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/06/14 15:39:56 [notice] 1#1: using the "epoll" event method
2024/06/14 15:39:56 [notice] 1#1: nginx/1.25.3
2024/06/14 15:39:56 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10) 
2024/06/14 15:39:56 [notice] 1#1: OS: Linux 6.5.0-27-generic
2024/06/14 15:39:56 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/06/14 15:39:56 [notice] 1#1: start worker processes
2024/06/14 15:39:56 [notice] 1#1: start worker process 22
2024/06/14 15:39:56 [notice] 1#1: start worker process 23
2024/06/14 15:39:56 [notice] 1#1: start worker process 24
2024/06/14 15:39:56 [notice] 1#1: start worker process 25
2024/06/14 15:39:56 [notice] 1#1: start worker process 26
2024/06/14 15:39:56 [notice] 1#1: start worker process 27
2024/06/14 15:39:56 [notice] 1#1: start worker process 28
2024/06/14 15:39:56 [notice] 1#1: start worker process 29
2024/06/14 15:39:56 [notice] 1#1: start worker process 30
2024/06/14 15:39:56 [notice] 1#1: start worker process 31
2024/06/14 15:39:56 [notice] 1#1: start worker process 32
2024/06/14 15:39:56 [notice] 1#1: start worker process 33
2024/06/14 15:39:56 [notice] 1#1: start worker process 34
2024/06/14 15:39:56 [notice] 1#1: start worker process 35
2024/06/14 15:39:56 [notice] 1#1: start worker process 36
2024/06/14 15:39:56 [notice] 1#1: start worker process 37
2024/06/14 15:39:56 [notice] 1#1: start worker process 38
2024/06/14 15:39:56 [notice] 1#1: start worker process 39
2024/06/14 15:39:56 [notice] 1#1: start worker process 40
2024/06/14 15:39:56 [notice] 1#1: start worker process 41

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Tasks

• Pipeline failure has been resolved

Please read the Iron Bank Documentation for more info