Problem with updating Fluent-bit from 1.9.9 to 2.0.3

The labels for this issue have been adjusted because one or more labels were identified as being incorrectly set.

assigned to @jbelleza and @sean.melissari

added 1 deleted label

Poking around at this issue I believe the issue is with permissions on several directories (possibly changed in the artifacts being pulled here). For context, Big Bang runs fluentbit as root. I'm seeing discrepancies in permissions with the latest image on /home/fluent-bit.

Permissions on an older IB Fluentbit (1.9.9):

bash-4.4# ls -la
total 20
drwxrwxr-x 5 fluent root 4096 Oct 26 02:55 .
drwxr-xr-x 1 root   root 4096 Nov  3 16:51 ..
drwxr-xr-x 2 root   root 4096 Oct 26 02:54 bin
drwxr-xr-x 2 root   root 4096 Oct 26 02:55 etc
drwxrwxr-x 2 fluent root 4096 Oct 26 02:49 log

Permissions on upstream Fluentbit 2.0.3 (in this case we are in /fluent-bit):

/fluent-bit # ls -la
total 20
drwxr-xr-x    5 root     root          4096 Oct 28 12:55 .
drwxr-xr-x    1 root     root          4096 Nov  3 16:59 ..
drwxr-xr-x    2 root     root          4096 Oct 28 12:55 bin
drwxr-xr-x    2 root     root          4096 Oct 28 12:55 etc
drwxr-xr-x    2 root     root          4096 Oct 28 12:45 log

Permissions on Ironbank latest (2.0.3):

bash-4.4# ls -la
total 20
drwxr-x--- 5 fluent-bit fluent-bit 4096 Nov  3 00:40 .
drwxr-xr-x 1 root       root       4096 Nov  3 00:40 ..
drwxr-x--- 2 fluent-bit fluent-bit 4096 Nov  3 00:40 bin
drwxr-x--- 2 fluent-bit fluent-bit 4096 Nov  3 00:40 etc
drwxr-x--- 2 fluent-bit fluent-bit 4096 Nov  3 00:32 log

Ownership and group/other permissions are off it appears like. I'm not sure if there are other directories this affects but these are the obvious ones.

There has been a semi-recent change in VAT adding some new CCEs via OpenSCAP that would be related to what you are seeing here. Oddly enough, this works fine locally and you are able to run it. Perhaps you guys are running a config that is grabbing logs that the fluentbit user does not have permissions to read?

I'm seeing the same behavior as you I think - docker runing the image it starts up fluentbit fine. But, when deployed via the helm chart it errors out starting up in Kubernetes. We're actually running the pod as root, so it's surprising that the error is "permission denied" (you'd assume root can do everything). We could explore changing our runAsUser to fluent-bit but this may have other side effects and I was hoping the easy solution would be to maintain parity with how things worked before.

I am wondering if the stat bin/fluent-bit: permission denied: unknown is misleading a little bit. The user has the ability to run the executable for sure, as we have both confirmed in docker. What flavor of kubernetes are you running on? RKE, RKE2, Vanilla, etc?

k3d (k3s in docker) currently, but I can run it on RKE2 or something else if you think there's a reason the distro could be causing issues.

84e5d10d

I think you're correct, this change was what caused the issues we are seeing now since it adjusted the chowns/directories. I'll play around with some changes and see if I can get it functioning with this.

From a quick test adding this to the dockerfile was enough to get passed this initial issue with permission denied on the stat:

RUN chmod -R +x /home/fluent-bit

I might be off base on this but it seems like the distro/scheduler reads the permissions on the file/directory and since root is not in the user/group and it isn't group executable it errors out.

Regardless of the why, that got me to another error so still more permissions problems

[2022/11/03 18:43:33] [error] could not open configuration file, aborting.

I will get k3d setup locally and give it a shot. I have as suspicion that the config could be mounted with a different default mode and that may help move the needle more.

What version of k3d are you running? I would like to emulate your setup as much as possible.

k3d version v5.4.6
k3s version v1.24.4-k3s1

Our pipeline also reproduced the initial issue with the IB image, which is on:

k3d version v5.4.1
k3s version v1.23.4-k3s1

Perfect, thank you

Would there be resistance to simply reverting the changes in 84e5d10d ? Fluentbit is almost always required to run as root since it needs to have permission to read logs on the node/host, which could be written with a variety of users and permissions. There may be alternative solutions here, such as the chmod I did above, but I wonder if we're chasing fixes that aren't worth the effort given that the pod runs as root anyways.

I am inclined to agree with you that reverting back would be the best option at this point, at least for now. I do believe there would be some resistance but we have a good reason here to do so. Let me find the correct person to pester to make sure it does not get rejected and I will report back. I would like to see if we can avoid the requirement for running as root, but having proof of it working in a cluster rootless should be a requirement of that.

Here's one issue that lays out most of the problem - https://github.com/fluent/fluent-bit/issues/872 - I'm not sure if this is well documented outside of people asking this question upstream but I check with the BB team to see if there's other places we've gone through the investigation.

Yeah I saw that post, specifically the comment here: https://github.com/fluent/fluent-bit/issues/872#issuecomment-827763207 which could be a good middle ground for security between BB and IronBank. As they state, its not the same as rootless but its not exactly root either.

Yeah, we actually do the drop: - ALL by default here. I noticed we don't add back the capability listed which is interesting, but didn't seem to change any of the errors I'm seeing.

Hey @micah.nagel would you be able to run this in your cluster with the following image: registry1.dso.mil/ironbank-staging/opensource/fluent/fluent-bit:ibci-1069265

This is an image that is in staging that will run as root. I would like to see if that will resolve the issue before pushing to master.

Please do keep in mind it runs as root, I know this is going to run on a local system just want to make sure the security implications are understood.

@hanafin.jacob unfortunately I don't have permissions to pull from staging. I may be able to load the tar from the pipeline directly, will get back to you on that.

Looks like that works well! Not seeing any errors in the pod logs and fluentbit is scraping host logs as expected.

Perfect, thats what I needed to hear. I am going to continue the process getting it pushed to master.

Is it possible you could share some fluentbit configs?

If you want to follow our deployment exactly the helm chart is here - https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit/-/tree/renovate/ironbank - the only modification you'd need to make would be to the image registry/repo to point to IB. Outputs might not work smoothly unless you deploy from Big Bang itself...but that might be overkill for testing this issue since the pod won't even start up.

Thank you, I am reviewing it now.

mentioned in merge request !177 (merged)

mentioned in merge request !178 (merged)

added 1 deleted label and removed 1 deleted label

added 1 deleted label

assigned to @alfontaine

mentioned in commit f56816da

closed

Problem with updating Fluent-bit from 1.9.9 to 2.0.3

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Possible fixes

Tasks

Child items 0

Activity