hardening_manifest.yaml testing and validation
Our change to hardening_manifest.yaml usage will require extensive testing and validation. Putting this ticket in as a placeholder so these activities can be tracked.
Currently using the pipeline-test-project to test. A dev-james branch has been made on each repo with .gitlab-ci.yml configuration for the testing branches.
Test plan
-
Update all repos to point CI to the feature branch -
Add a project (jenkins) using renovate.json to the pipeline test project and grant @renovate-bot access to the repo -
Run the migration script pointing it at the pipeline-test-projectpython3 ./scripts/hardening_manifest_yaml/migration.py \ --repo1-token="ironbank-bot-personal-access-token" \ --dccscr-whitelists-branch=pipeline-test-project \ --dccscr-whitelists-path=opensource/pipeline-test-project \ --start-branch=dev-james \ --force=true -
Merge some of the MR's into developmentto test behavior with hardening_manifest.yaml present and missing -
Merge some of the MR's into masterto test behavior with hardening_manifest.yaml present and missing -
Run the trigger script pointing it at pipeline-test-project -
Wait ~24h for @ironbank-bot to review the MRs. -
Wait ~24h for @renovate-bot to review the repo. -
[ ] Merge the changes in and then delete the deprecated fields out of the greylist
- Decided to not support deprecation of these fields for this current iteration of the pipeline. There are currently one stage that will break
csv-output(justifier.py)
Test matrix
Check that the entire pipeline runs successfully in each of these scenarios:
| master | development | feature branch | |
|---|---|---|---|
| Before MR (compatibility mode) No hardening_manifest.yaml in repo |
|||
| After hardening_manifest MR |
-
Migrated renovate.json Test if the regexManagers allow renovate to correctly parse hardening_manifest.yaml (downgrading the version to test if needed) -
ironbank-bot doesn't make comments on generated MRs [x] Test duplicate--labels (what does the error look like, or does our autogenerated one silently replace it?)-
Test duplicate --build-args What ifBASE_REGISTRYexists in hardening_manifest.yaml? -
Change tag in feature branch Test bumping the tag in a feature branch
(Please don't look at the markdown/HTML of this issue, it is horrifying)