Falco Research Spike (Deployment, Updating, Integrating)
Related to #939 (closed).
This spike should evaluate deployment, updates, and integration of Falco within Big Bang.
Open Source
Currently, BigBang core only supports Twistlock for runtime defense. Since Twistlock is a commercial product, this effort will explore Falco as an alternative.
Twistlock
Twistlock currently has several deficiencies
- Operator issues &115 (closed)
- Operator can't install policies
- Operator can't install defenders pointing to external Twistlock console.
Assessment
This effort will assess Falco's ability to:
- Deploy Falco via GitOps
- Deploy "defenders" via GitOps
- Deploy "defenders" to external falco (hub/spoke)
- Does Falco work in containerized (k3d) environments for testing
Activity
added to epic &138 (closed)
changed milestone to %1.22.0
changed iteration to Big Bang Iterations Nov 16, 2021 - Nov 29, 2021
added priority5 label
added teamcore/security label
we can split this issue into two research spikes
- evaluate the ease of deployment / updating / integrating Falco into Big Bang
- evaluate / compare features between twistlock and Falco to understand the delta between the tools
Edited by Gabe Scarberryadded customerpartybus label
removed customerpartybus label
assigned to @micah.nagel
added statusdoing label
mentioned in issue #939 (closed)
Using https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/falco for investigation.
Initial upstream chart pulled from https://github.com/falcosecurity/charts/tree/falco-1.16.1/falco
Edited by Micah Nagel-
They also have a "falco-exporter" chart - https://github.com/falcosecurity/charts/tree/falco-1.16.1/falco-exporter - which contains a prometheus exporter and grafana dashboard.
-
Was able to get Falco deployed rather easily on k3d. You need to add some extra volume mounts to get it going.
sudo k3d cluster create --servers 1 --agents 3 --api-port 6443 --k3s-server-arg "--disable=traefik" --k3s-server-arg \ "--disable=metrics-server" --k3s-server-arg "--tls-san=$(curl -s http://checkip.amazonaws.com)" \ -v /etc/machine-id:/etc/machine-id -v /var/run/containerd/containerd.sock:/run/containerd/containerd.sock \ -v /var/run/docker.sock:/var/run/docker.sock -v /boot:/boot -v /lib/modules:/lib/modules \ -p 80:80@loadbalancer -p 443:443@loadbalancerNote that the main differences are extra volume mounts for:
- containderd sock
- docker sock
/boot/lib/modules
k3d nodes already have the other folders that falco scans/looks through (such as
/usrand/proc).Edited by Micah Nagel -
There is some weirdness I'm seeing with this this morning...looks like:
- k3d won't automatically update the kubeconfig sometimes
(likely one of these volume mounts messing with it?)jk I'm dumb, sudo. - sometimes the falco pods won't start up and complain about missing/inaccessible file
/host/dev/falco0(which maps to the k3d/dev/falco0)
Trying to trace down these issues...might be
sudoneeded on thek3d createcommand...Edited by Micah Nagel - k3d won't automatically update the kubeconfig sometimes
-
Added
sudoto the above k3d create command. That seems to give consistency in things starting up, although the kubeconfig doesn't get updated for the primary user since thek3d createis run as root (therootuser kubeconfig does get updated).k3d kubeconfig get k3s-defaultgets the kubeconfig which is an easy workaround.
-
Falco definitely seems to work well in k3d. In terms of deployment...
- falco (main chart): "Falco is a Cloud Native Runtime Security tool designed to detect anomalous activity in your applications. You can use Falco to monitor runtime security of your Kubernetes applications and internal components." This is the equivalent of defenders from Twistlock. Each "falco" pod runs on a single node and monitors/logs activity.
- falco sidekick (sub chart): "A simple daemon for connecting Falco to your ecosystem. It takes a Falco's events and forward them to different outputs in a fan-out way." This allows output in a ridiculously large amount of formats (see full list here). For initial testing purposes I deployed with the built in UI, which provides two simple views:
These two views give a basic dashboard that shows an "at a glance" view of everything going on + an event log with history of everything (up until the retention point).
Will attempt to deploy with some of the additional integrations to see how easy they are to setup (targeting Elastic, Mattermost, and Prom/Grafana to test).
Edited by Micah Nagel -
Mattermost integration is easy to setup. Only required parameter is a webhook URL. Additional config options for filtering what is posted, customizing the bot name/image, even message format if you want extremely customized output.
Edited by Micah Nagel
-
From some initial investigation the hub/spoke model is supported to an extent:
- Every cluster would need to deploy falco + sidekick
- The sidekick UI would only show the details for the cluster it is on
- You can setup config/integrations to hook up the sidekick to anything inside or outside of the cluster
I haven't found a way to connect a sidekick instance to multiple clusters with falco. I might just not have found it yet though...I'm trying to investigate if we can connect a sidekick UI to other sidekicks but there doesn't seem to be much config for the UI.
Edited by Micah Nagel -
I think this spike is good to go. The above comments have some scattered details so I have provided a rundown of each investigation point below, along with links to comments that have more detail.
Deploy Falco via GitOps
Falco provides a good helm chart for both the main falco deployment and sidekick deployment. Everything can be done via values.
- Rules can be added via new yaml files or via
customRulesvalue which gets provided to the falco instances (this was one of the biggest downsides to Twistlock) - New rules don't overwrite the default set/what was set previously (unless you change values, all in customer control)
- Daemonset has checksums to ensure defenders get the latest rules from configmaps when the values change
Deploy "defenders" via GitOps
Falco comparisons to Twistlock are tricky to navigate with the "console/defender" model of Twistlock. But at its core, yes the equivalent to a "defender" can easily be deployed via GitOps.
- Falco (the "main" component and main chart): Equivalent to the defenders for Twistlock. Daemonset that runs on your nodes and monitors everything for rule violations.
- Falco Sidekick (a subchart): Somewhat equivalent to the Twistlock console, but that comparison is very loose. Falco Sidekick does come with a basic UI that you can enable, but the primary purpose is for forwarding the information to external integrations (Mattermost, Elastic, Loki, Grafana, Alertmanager, and tons of other formats - including message queues, object storage, and serverless things). See screenshots/details of the sidekick integrations here.
Deploy "defenders" to external falco (hub/spoke)
More detail in this comment - but the hub/spoke model is loosely supported. Each cluster would need a falco daemonset and falco sidekick deployment but you could have the falco sidekicks utilize the same config/integration so that they all forward information to the same mattermost instance under the same webhook url (for example).
Does Falco work in containerized (k3d) environments for testing
Yes yes yes! I did all of my testing on k3d. I'm sure the information is more or less relevant and helpful when you're running on docker containers rather than a "real" host - but it definitely works to monitor and forward information. More details on how to deploy here - it requires running the
k3d cluster createas root to give falco proper access perms into some of the folders + adding additional volume mounts for each k3d node (-vparams to the create command).
Overall big
Edited by Micah Nagel - Rules can be added via new yaml files or via
added statusreview label and removed statusdoing label
removed statusreview label
