Update Fluentbit to 1.8.1

New IB image available.

Make sure to pull in any upstream chart updates (if applicable).

Definition of done:

• Image version(s) updated in chart/values.yaml
• Chart version bumped (match upstream chart, reset to bb.0 if the upstream version changes)
• Changelog entry added, note the update to image, any additional updates
• Clean install tested and validated working (see things to test)
• Upgrade tested and validated working (install newest tagged release, upgrade to your branch, then run through things to test)
• Pipeline passing on package MR
• Package MR merged and new release tagged
• BB MR opened, pipeline passing
• BB MR merged (no checkbox, BB MR should auto-close this)

Things to test:

• Install BB with core enabled
• Validate all pods come up healthy
• Validate that logs are being passed to elastic/kibana (for the upgrade ensure logs come in post-upgrade in addition to the ones already there)

added to epic &63 (closed)

added fluentbit priority6 + 1 deleted label

set weight to 2

changed the description

changed iteration to Big Bang Iterations Jul 27, 2021 - Aug 9, 2021

set weight to 3

assigned to @mark.sanchez

added statusdoing label

marked the checklist item Image version(s) updated in chart/values.yaml as completed

marked the checklist item Chart version bumped (match upstream chart, reset to bb.0 if the upstream version changes) as completed

• Updated chart to pull in changes the upstream chart using kpt pkg update chart@fluent-bit-0.16.1 --strategy alpha-git-patch
• resolved merge conflicts
• seeing issues with some pods sometimes when deploying
• not sure why all pods wouldnt have the same issue

logging-fluent-bit-lpq6q         2/2     Running            0          37m
opa-collector-b6b947c96-94qct    2/2     Running            1          37m
logging-fluent-bit-vmznc         2/2     Running            0          37m
logging-ek-es-data-1             2/2     Running            0          37m
logging-ek-es-data-0             2/2     Running            0          37m
logging-ek-es-master-0           2/2     Running            0          37m
logging-ek-kb-7957d64669-lgfxm   2/2     Running            0          36m
logging-fluent-bit-br5sd         1/2     CrashLoopBackOff   10         37m
logging-fluent-bit-s2fsr         1/2     CrashLoopBackOff   11         37m

  fluent-bit:
    Container ID:   containerd://19c0e4774e75e35a84df7b5407d5791e5f99ef5c007ef76d2c82e88193183f66
    Image:          registry1.dso.mil/ironbank/opensource/fluent/fluent-bit:1.8.2
    Image ID:       registry1.dso.mil/ironbank/opensource/fluent/fluent-bit@sha256:cbb8a4eb54b94d366044548eddd54753698698a0f876891625c81558f99ccd8d
    Port:           2020/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    139

  Warning  Unhealthy    32m                    kubelet            Readiness probe failed: Get "http://10.42.3.120:15020/app-health/fluent-bit/readyz": dial tcp 10.42.3.120:15020: connect: connection refused
  Warning  Unhealthy    32m (x8 over 32m)      kubelet            Readiness probe failed: Get "http://10.42.3.120:15021/healthz/ready": dial tcp 10.42.3.120:15021: connect: connection refused
  Normal   Pulled       31m                    kubelet            Successfully pulled image "registry1.dso.mil/ironbank/opensource/fluent/fluent-bit:1.8.2" in 1.170517721s
  Warning  BackOff      7m56s (x110 over 30m)  kubelet            Back-off restarting failed container
  Normal   Pulling      2m50s (x11 over 32m)   kubelet            Pulling image "registry1.dso.mil/ironbank/opensource/fluent/fluent-bit:1.8.2"

removed statusdoing label

unassigned @mark.sanchez

assigned to @mark.sanchez

added statusdoing label

Ran on Fresh EC2/Cluster and no issues.

Logs show up in Kibana

Moving on to test upgrade.

marked the checklist item Clean install tested and validated working (see things to test) as completed

Testing upgrade

All pods came up successfully

Logs still showing up in kibana

After few min 1 pod went into crashloop backoff

[2021/08/02 14:50:29] [engine] caught signal (SIGSEGV)
#0  0x5ca7fe            in  mk_list_is_empty() at lib/monkey/include/monkey/mk_core/mk_list.h:100
#1  0x5caa4c            in  is_healthy() at src/http_server/api/v1/health.c:75
#2  0x5cb051            in  cb_health() at src/http_server/api/v1/health.c:266
#3  0x9af791            in  thread_cb_init_vars() at lib/monkey/mk_server/mk_http_thread.c:69
#4  0x9ba986            in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#5  0xffffffffffffffff  in  ???() at ???:0

changed title from Update Fluentbit to 1.8.2 to Update Fluentbit to 1.8.1

Per the truckload of comments above Mark confirmed that 1.8.1 image does not suffer from the same fate that the 1.8.2 and 1.8.3 images available both from Ironbank and upstream.

Updating issue to be update to version 1.8.1

Didnt expect this one to be so rough lol

marked the checklist item Changelog entry added, note the update to image, any additional updates as completed

marked the checklist item Upgrade tested and validated working (install newest tagged release, upgrade to your branch, then run through things to test) as completed

Pods running up after upgrade and logs showing up

marked the checklist item Pipeline passing on package MR as completed

Validated stayed running overnight for good measure.

added Affinity/AntiAffinity label

removed Affinity/AntiAffinity label

added statusreview label and removed statusdoing label

added statusdoing label and removed statusreview label

marked the checklist item Package MR merged and new release tagged as completed

created merge request !731 (merged) to address this issue

mentioned in merge request !731 (merged)

marked the checklist item BB MR opened, pipeline passing as completed

added statusreview label and removed statusdoing label

closed with merge request !731 (merged)

mentioned in commit cb709dbe

mentioned in issue #699 (closed)

removed statusreview label