Twistlock system custom rule IDs are not consistent - breaks policies during upgrades

Twistlock system custom rule IDs are not consistent between different installations. Some rules (especially Waas) are in a different order, some are named differently, and some system custom rules only exist if they were installed with a particular version of PCC.

Here is a screenshot showing the custom rules that are created by the system. Note, there is a separate tab for the Waas rules. System rules come from the vendor and often accompany a new release.

The bottom line is that you cannot depend on the system custom rules IDs to be consistent for deploying policies. I have seen situations where the runtime IDs were incorrect and results in a bad policy.

Solution:

Don't use the system rules. Instead clone them and use the clones for policies. The user custom rule policy ID numbers need to be translated (to include their usages in policies) during install time. I have already written code to do this as a part of effort to backup and restore runtime configurations between multiple environments.

We should discuss this and Twistlock CaC in general further.

@phillip.warner Is there a way to uniquely identify the rules so when the ID/name changes we know its the same rule? By cloning, we may end up with several copies of the same rule if the ID/name changes between releases. Any ideas on how to prevent that?

Also, do you know which part of our init might hit this? I don't remember if we used ID or name to identify rules or if we just use attributes (e.g. severity).

@michaelmcleroy , you only clone the system rules when you first build the set in code. From then on it is best to wipe out all of the custom rules and add back the ones defined in your code. However, there are dependencies on this with the runtime policies and alert profiles.

The most repeatable CaC requires tearing down the configuration and building it back up. Soon, I'd like to upstream the backup/restore code to this project along with a simplified configuration based on the work I've been doing with Party Bus.

@phillip.warner Shot in the dark, but have you pushed this code anywhere?

added teamXForce label

removed teamXForce label

added teamSecurity & Compliance label

added twistlock label

@phillip.warner this issue has been inactive for 30 days and is being labelled as stale. If this issue is still required please take action by removing the stale label and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken this issue will be auto closed in 60 days.

added stale label

@phillip.warner this issue has been inactive for 60 days and is being labelled as marked-for-auto-close. If this issue is still required please take action by removing the stale and marked-for-auto-close labels and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken this issue will be auto closed in 30 days.

added marked-for-auto-close label

removed marked-for-auto-close label

removed stale label

added priority6 label

added kindfeature label

@phillip.warner this issue has been inactive for 30 days and is being labelled as stale. If this issue is still required please take action by removing the stale label and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken this issue will be auto closed in 60 days.

added stale label

@phillip.warner this issue has been inactive for 60 days and is being labelled as marked-for-auto-close. If this issue is still required please take action by removing the stale and marked-for-auto-close labels and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken this issue will be auto closed in 30 days.

added marked-for-auto-close label

assigned to @dax.mcdonald

removed marked-for-auto-close label

removed stale label

added statusdoing label

set weight to 3

added statusblocked label and removed statusdoing label

"Twistlock system custom rules" don't seem specific enough for me.

There are several rule endpoints available. https://pan.dev/compute/api/get-policies-firewall-app-app-embedded/ is one, https://pan.dev/compute/api/get-custom-rules/ there are also these rules.

My current understanding is that the rules referenced here likely the compliance policy custom id rules but I moved this issue to blocked until I can confirm

@dax.mcdonald , I am referring to the runtime policies custom rule IDs. That pool of IDs is shared across all of the different runtime policies (host, container, Waas, etc).

My solution works and has been in use for a long time, but it is a run on demand script. Some small changes can make it so it could run as part of the Big Bang init. I would want to include a simplified custom set of collections and policies as well (simplified compared to production deployments, but much more complex than what is built into big bang currently).

changed the description

Added an image to the description. The rule IDs themselves are only seen once you export the json policies and rules; you will not see them in the console.

added statusdoing label and removed statusblocked label

@phillip.warner That is super helpful. https://pan.dev/compute/api/32-03/get-custom-rules/ This looks to be the api to pull all these rules. I haven't done an upgrade of Twistlock yet so I haven't re-produced the behavior but it sounds like the exported rules _id field changes.

As an example of one of the rules:

 {
    "name": "Twistlock Labs - GKE - exec or attach to a pod",
    "_id": 8,
    "type": "kubernetes-audit",
    "script": "jpath(\"protoPayload.methodName\") = \"io.k8s.core.v1.pods.exec.create\" or jpath(\"protoPayload.methodName\") = \"io.k8s.core.v1.pods.attach.create\"",
    "description": "Audit any exec or attach event to a pod on GKE",
    "message": "Exec or attach to a pod detected on GKE",
    "attackTechniques": [
      "execIntoContainer"
    ],
    "owner": "system",
    "modified": 1718729444,
    "minVersion": "",
    "vulnIDs": []
  },

This _id field may change on upgrade?

@phillip.warner Maybe I'm missing something here but it looks like what we care about here is the script field.

So we could have the init script check to make sure a given list of rules exists in an installation. We could avoid relying on the id fields entirely in that case.

The BB policies have use specific IDs under the hood, because that is how twistlock policies reference them. Look for similar script fields is not enough. The other portions are important as well, and using the wrong ids in the policies will result in policies that may not be broken (i.e., they may install), but they will not work right.

Scripts for testing posted at https://repo1.dso.mil/big-bang/product/packages/twistlock/-/tree/wip-scripts-for-testing/chart/scripts/contrib These are what I have been using to manage P1 deployments for a long time.

changed weight to 7 from 3

added statusblocked label and removed statusdoing label

unassigned @dax.mcdonald

Going to move to this to blocked until we solve some higher level questions:

1. This backup and restore code is destructive and to run unattended it would need some additional checks. ie, if the init job is killed during the restore it may not restore all the custom rules but will run through an entire delete flow.
2. We should raise this with Palo Alto to get their feedback on why their Backup/Restore flow doesn't handle this use case.
3. If we decide to proceed here, we should break this out into smaller issues to handle backup and continous upgrades as separate concerns. This issue can arise either as part of a CI/CD pipeline where init is always set or when needing to migrate a set of Twistlock custom rules to another system.

@phillip.warner this issue has been inactive for 30 days and is being labelled as stale. If this issue is still required please take action by removing the stale label and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken this issue will be auto closed in 60 days.

added stale label

@phillip.warner this issue has been inactive for 60 days and is being labelled as marked-for-auto-close. If this issue is still required please take action by removing the stale and marked-for-auto-close labels and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken this issue will be auto closed in 30 days.

added marked-for-auto-close label

added community-contribution label

this still needs to be fixed. Palo Alto said they would not change it when I asked them to a while ago. I could try again...for now, I do think we can work to at least get some of the backup/restore capability integrated into Big Bang.

Translating the IDs before installing the policies is not difficult and is already implemented in the restore scripts in the wip branch previously mentioned.

removed marked-for-auto-close label

removed stale label

assigned to @daxmc99

Hi @phillip.warner, I missed this update. Will reach out over mattermost.

changed iteration to Big Bang Iterations Oct 15, 2024 - Oct 28, 2024

removed iteration Big Bang Iterations Oct 15, 2024 - Oct 28, 2024

changed iteration to Big Bang Iterations Oct 29, 2024 - Nov 11, 2024