UNCLASSIFIED - NO CUI

Problem with updating Fluent-bit from 1.9.9 to 2.0.3

Summary

Upgrading from fluent-bit 1.9.9 to 2.0.3 results in fluent-bit pods stuck in a CrashLoopBackOff and empty logs

Steps to reproduce

run Big Bang with Fluent-bit and dependencies enabled

What is the current bug behavior?

a kubectl describe of the pods gives logs that indicate an error starting

  Warning  Failed     44m (x3 over 44m)    kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "bin/fluent-bit": stat bin/fluent-bit: permission denied: unknown

What is the expected correct behavior?

Pods that start and run (which happens when I pull the image from cr.fluentbit.io/fluent/fluent-bit instead of registry1

Relevant logs and/or screenshots

logging          pod/logging-fluent-bit-2scp2                                    1/2     CrashLoopBackOff   13 (119s ago)   43m
logging          pod/logging-fluent-bit-54fcb                                    1/2     CrashLoopBackOff   13 (116s ago)   43m
logging          pod/logging-fluent-bit-ckr2f                                    1/2     CrashLoopBackOff   13 (113s ago)   43m
logging          pod/logging-fluent-bit-dk5qg                                    1/2     CrashLoopBackOff   13 (93s ago)    43m
  Normal   Pulled     44m                  kubelet            Successfully pulled image "registry1.dso.mil/ironbank/opensource/fluent/fluent-bit:2.0.3" in 8.504078615s
  Normal   Created    44m (x3 over 44m)    kubelet            Created container fluent-bit
  Warning  Failed     44m (x3 over 44m)    kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "bin/fluent-bit": stat bin/fluent-bit: permission denied: unknown
  Normal   Pulled     44m (x3 over 44m)    kubelet            Container image "registry1.dso.mil/ironbank/opensource/fluent/fluent-bit:2.0.3" already present on machine
  Warning  BackOff    20s (x231 over 44m)  kubelet            Back-off restarting failed container

Possible fixes

Unknown

Tasks

  • Bug has been identified and corrected within the container
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information