Gatekeeper default policy blocking twistlock and monitoring

The default configuration of the gatekeeper no-host-namespace policy (k8spsphostnamespace.constraints.gatekeeper.sh) appears to be blocking the deployment of monitoring and twistlock components in BigBang 1.13.1.

~ kubectl get k8spsphostnamespace.constraints.gatekeeper.sh -A no-host-namespace -o yaml
...
violations:
 - enforcementAction: deny
    kind: Pod
    message: 'Sharing the host namespace is not allowed: monitoring-monitoring-prometheus-node-exporter-zwp5h'
    name: monitoring-monitoring-prometheus-node-exporter-zwp5h
    namespace: monitoring
  - enforcementAction: deny
    kind: Pod
    message: 'Sharing the host namespace is not allowed: twistlock-defender-ds-5gsbq'
    name: twistlock-defender-ds-5gsbq
    namespace: twistlock
...

Suggest excluding those namespaces from enforcement by default here: https://repo1.dso.mil/platform-one/big-bang/apps/core/policy/-/blob/main/chart/values.yaml#L227

With:

  noHostNamespace:
    match:
      excludedNamespaces:
        - monitoring
        - twistlock

added kindbug label

Looks like twistlock is also running into the host-network policy (k8spsphostnetworkingports.constraints.gatekeeper.sh) as well.

kubectl get k8spsphostnetworkingports.constraints.gatekeeper.sh host-networking -o yaml
...
  violations:
  - enforcementAction: deny
    kind: Pod
    message: 'The specified hostNetwork and hostPort are not allowed, pod: twistlock-defender-ds-5gsbq.
      Allowed values: {"hostNetwork": false}'
    name: twistlock-defender-ds-5gsbq
    namespace: twistlock
...

Thanks. This was an oversight that we will correct next release.
To workaround the issue set the following in Big Bang:

gatekeeper:
  values:
    violations:
      noHostNamespace:
        match:
          excludedNamespaces:
          - monitoring
          - twistlock

changed milestone to %1.16.0

changed iteration to Big Bang Iterations Aug 24, 2021 - Sep 6, 2021

added gatekeeper priority6 labels

Thanks @michaelmcleroy, I think that might be missing a yaml level - but I can confirm that approach resolved the issue.

We updated our bigbang values with:

gatekeeper:
  values:
    violations:
      noHostNamespace:
        match:
          excludedNamespaces:
          - monitoring
          - twistlock
      hostNetworking:
        match:
          excludedNamespaces:
          - twistlock
          - monitoring

Everything came up as expected

Good catch. I'll edit my comment to avoid in future confusion

@ablanchard hostNetworking also causes denial?

Correct, logs are linked above in the first comment.

set weight to 2

unassigned @ablanchard

assigned to @jennifer.kays

added statusdoing label

mentioned in merge request !824 (closed)

created merge request !824 (closed) to address this issue

Installed twistlock-defender using the twistlock UI. Successfully recreated reported twistlock-defender errors. For example: [no-host-namespace] Sharing the host namespace is not allowed: twistlock-defender-ds-rtwl5

FYSA, I believe this issue is resolved in gatekeeper MR !89.

Thanks for passing this along. We've moved gatekeeper exceptions into BB rather than the package so I don't think we'll merge that as is, but it gives us all the right exceptions we need.

Monitoring is resolved with this MR into BigBang https://repo1.dso.mil/platform-one/big-bang/bigbang/-/merge_requests/831

Wasn't properly addressed a couple weeks ago which is why monitoring hasn't had an exception listed. No longer required with above MR.

@jennifer.kays it looks like we'll need 4 exceptions here:

• hostNetworking: defenders need for ???, we should find the reason and document if possible
• noHostNamespace: attached to above
• restrictedTaint: defenders need to run on all nodes regardless of taint
• allowedHostFilesystem: defenders need to mount files from the nodes to monitor them for security

Since all of these exceptions are for the defenders we might be able to use the new parameters.excludedResources value instead of excluding the whole ns. For that you'll just need namespace/resource name where resource name matches with the name that is shown in the violation message (supports regex). Example, although this likely isn't accurate to what is needed, twistlock/defenders-.* would add an exception for things named defender-xyz defender-zxy in the twistlock ns.

Happy to sync if that gets confusing. We may have to just add the whole ns exception if user can modify the names of the created resources.

Thanks, @micah.nagel! I also noticed the hostNetworking and allowedHostFilesystem checks getting in the way of defender. I will have to look again for the restrictedTaint violation.

I've proven that excluding the twistlock namespace for noHostNamespace removes those violations. I'll go back and try it again with parameters.excludedResources.

One thing I've learned is I have to be patient when verifying results. I don't know if it's just twistlock, but the effects aren't showing up quickly!

For restrictedTaint you may not see it in normal testing since your nodes likely don't have taint. If you were to add taint to a node (see this doc) you should find that the defenders will not try to schedule on that node. Since the defenders are doing security monitoring for us we want them on every node regardless of status.

I haven't tested this one myself, but the way the violation is setup it would block Twistlock Defenders from that node, and we do want them.

assigned to @mark.sanchez

mentioned in merge request !852 (merged)

Merge request created: https://repo1.dso.mil/platform-one/big-bang/bigbang/-/merge_requests/852

added statusreview label and removed statusdoing label

mentioned in commit e46457ed

closed with merge request !852 (merged)

mentioned in commit f499ed44

removed statusreview label