Exec Format Error

Summary

With the latest tag 1.21 (sha256:63c63a34976a5413a4ff22df7fa4593b2e3b26a61db545353c691c1cba9abb8a), there is a "exec format error" that occurs when running pilot-agent. The previous build of the 1.21 tag that I pulled about 4 weeks ago, sha256:6afa7b83d0b10fc7b8314657698bd4e21189eb0a5e1560d29466d4f17f18495c, is working correctly.

Steps to reproduce

$ docker run --pull always --rm registry1.dso.mil/ironbank/tetrate/istio/proxyv2:1.21

and/or:

$ docker run --pull always --rm registry1.dso.mil/ironbank/tetrate/istio/proxyv2@sha256:63c63a34976a5413a4ff22df7fa4593b2e3b26a61db545353c691c1cba9abb8a

What is the current bug behavior?

Observe the following output. This occurs regardless of any CLI arguments passed to the container (e.g. version or help)

$ docker run --pull always --rm registry1.dso.mil/ironbank/tetrate/istio/proxyv2:1.21
1.21: Pulling from ironbank/tetrate/istio/proxyv2
Digest: sha256:63c63a34976a5413a4ff22df7fa4593b2e3b26a61db545353c691c1cba9abb8a
Status: Image is up to date for registry1.dso.mil/ironbank/tetrate/istio/proxyv2:1.21
exec /usr/local/bin/pilot-agent: exec format error

What is the expected correct behavior?

The container (and the pilot-agent binary ENTRYPOINT) runs properly.

Relevant logs and/or screenshots

I was able to copy the binary out of the image to inspect it. It's an AARCH64 (ARM64) binary running on a x86_64 machine.

$ CONTAINER_ID=$(docker create registry1.dso.mil/ironbank/tetrate/istio/proxyv2:1.21)
$ docker cp "$CONTAINER_ID":/usr/local/bin/pilot-agent /tmp/pilot-agent
Successfully copied 42.3MB to /tmp/pilot-agent
$ file /tmp/pilot-agent
/tmp/pilot-agent: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, Go BuildID=HH-HKS6Lrfj6IsqI8Ya6/4wTN0_p9R466mXg8jqve/e_LjKXW9uFtckyDCH_Ql/EJtk1CqDcB2UprI7A4oC, stripped

At some point between the two sha256 checksums I posted above, it seems that the binary was changed from x86_64 to aarch64. It may be related to commit 0e905038 where the base image was changed from Google distroless to Chainguard. I'm not sure why this image (proxyv2) was changed but the other 1.21 Istio images (operator, pilot, istioctl, install-cni) were not changed. Additionally, all the 1.22 Istio images also use Google distroless.

Possible fixes

Unsure. Possibly revert the base image change or package the x86_64 version of the binary instead of aarch64/arm64.

Tasks

• Bug has been identified and corrected within the container

Please read the Iron Bank Documentation for more info