Darrien Lee requested to merge 203-create-bbctl-all-versions-dashboard into main Oct 16, 2024

General MR

Summary

This dashboard should show all items in the cluster (bb, package, kubernetes, etc) and their version. This should be deployed as part of the bbctl helm chart.

#197 (closed) added in the helm chart, and #272 (closed) is further refining this chart. The chart will run bbctl version -U -A internally and output a logs-parseable format.

##NOTES:

This issue/ticket encompassed a lot of work done to get it to work as intended, and may have been out of scope of this ticket, or should have been broken out into its own issues. Regardless, here are some notes regarding what was accomplished and breakdown/explanation of the 18 points.

Worked on/Accomplished:

creating baseline template for future dashboard additions (2 points)
creating implementation for local grafana dashboard deployments (2 points)
k3d script changes (4 points)
- script changes caused promtail and neuvector issues, debugging, working with script owners, and updating script for EC2 fixes for these two deployments
- these script changes were found not to be cause of JSON not being seen by promtail/loki/grafana stack
fixed configuration files being unable to unmarshal credential files (1 point)
output issues prompted a new release from version 0.7.4 to 0.7.5 (3 point)
- worked on bbctl update release to 0.7.5 for potential JSON fixes (also testing/trying out goreleaser for first time)
- updated tags to reflect 0.7.5 release
- rebasing to new bbctl version
SPIKE: update 0.7.5 with potential JSON fixes were found to not be the cause of JSON not showing issues (4 points)
- promtail's intended behavior is found to tail files and split by newlines, meaning a raw JSON object will not be seen by our monitoring stack details here
- workaround found and implemented through echoing the output, may need new issue/ticket if we want to invest further time in getting this working
- Tried updating client.go and marshalling/unmarshalling files to add a newline, but promtail is still unable to find this output, threads here and here
- ~~updating JSON output marshaller formatting to fix promtail scraping~~ (1 point)
changed all other existing cronjob outputs to work in JSON mode using echo workaround (1 points)

Blockers:

k3d deployment script update broke my EC2 instances (metalLB, promtail, and neuvector)
- worked to circumvent these issues, as well as worked to fix the deployment script
JSON formatted output (which is needed for data manipulation) not showing up in k9s nor Grafana/Loki stack, but shows up in kubectl logs
- worked on 0.7.5 release (also testing goreleaser for bbctl) as a potential fix
- update 0.7.5 with potential fixes were found to not be the cause of JSON not showing issues