UNCLASSIFIED - NO CUI

hardening_manifest.yaml testing and validation

Our change to hardening_manifest.yaml usage will require extensive testing and validation. Putting this ticket in as a placeholder so these activities can be tracked.

Currently using the pipeline-test-project to test. A dev-james branch has been made on each repo with .gitlab-ci.yml configuration for the testing branches.

Test plan

  1. Update all repos to point CI to the feature branch

  2. Add a project (jenkins) using renovate.json to the pipeline test project and grant @renovate-bot access to the repo

  3. Run the migration script pointing it at the pipeline-test-project

     python3 ./scripts/hardening_manifest_yaml/migration.py \
         --repo1-token="ironbank-bot-personal-access-token" \
         --dccscr-whitelists-branch=pipeline-test-project \
         --dccscr-whitelists-path=opensource/pipeline-test-project \
         --start-branch=dev-james \
         --force=true
  4. Merge some of the MR's into development to test behavior with hardening_manifest.yaml present and missing

  5. Merge some of the MR's into master to test behavior with hardening_manifest.yaml present and missing

  6. Run the trigger script pointing it at pipeline-test-project

  7. Wait ~24h for @ironbank-bot to review the MRs.

  8. Wait ~24h for @renovate-bot to review the repo.

  9. [ ] Merge the changes in and then delete the deprecated fields out of the greylist

  • Decided to not support deprecation of these fields for this current iteration of the pipeline. There are currently one stage that will break csv-output (justifier.py)

Test matrix

Check that the entire pipeline runs successfully in each of these scenarios:

master development feature branch
Before MR (compatibility mode)
No hardening_manifest.yaml in repo
  •  
  •  
  •  
After hardening_manifest MR
  •  
  •  
  •  
  • Migrated renovate.json Test if the regexManagers allow renovate to correctly parse hardening_manifest.yaml (downgrading the version to test if needed)
  • ironbank-bot doesn't make comments on generated MRs
  • [x] Test duplicate --labels (what does the error look like, or does our autogenerated one silently replace it?)
  • Test duplicate --build-args What if BASE_REGISTRY exists in hardening_manifest.yaml?
  • Change tag in feature branch Test bumping the tag in a feature branch

(Please don't look at the markdown/HTML of this issue, it is horrifying)

Edited by James Petersen