Pipeline improvements to support Stargate

We need to make some adjustments to our artifacts so that they can be better consumed by our customers. These changes include the following updates to the tarball that can be downloaded from the Iron Bank:

1. Adding the justification column to the CSVs
   • #189 (closed) Before deprecating the XLSX file, the justifier must operate on CSVs instead of XLSX files
2. Create a new detached signature of the entire tarball. This file will be uploaded to S3 beside the tarball.
3. Add a sha256 docker reference (e.g. registry1.dso.mil/ironbank/redhat/ubi/ubi8@sha256:a5e5632ee49c9b9a7ff9be8ae4b0cd1ed0f2e568826e99eea67cf198e5e7eff8 to an image field in the documentation.json file.
4. The root directory of the tarball is reports. Consider something like ubi8-8.3-reports. This is not high priority.
5. #152 (closed) Produce documentation for this BOE
6. Keep the XLSX file and any deprecated files as-is for now, because they may be is currently in use and any downstream customers may depend on them. Users that don't want xlsx files may delete it themselves for now.

Proposed BOE/schema/api example

• Adding new files to the archive are NOT considered breaking changes. Removing files IS a breaking change
  • Deprecated files in the archive may be removed in the future.
• Adding columns to CSV files are NOT considered breaking changes. Removing columns from CSV files IS a breaking change.
• Adding new fields to documentation.json is NOT considered a breaking change.
• The root directory of the archive, reports, may be renamed in the future. All files in the archive will still be in one root directory.
• The contents of the anchore, openscap and twistlock are the output of the respective tools. The filenames inside these directories are reasonably stable, but the contents of these files are unspecified and are included as-is from the tools.
• The files oscap.csv, oval.csv and the openscap/ directory are omitted if scanning is not performed with OpenSCAP. Scratch and other non-UBI images are not scanned with OpenSCAP.

Cryptographic signature/chain of trust/validation

1. ubi8-8.3-reports-signature.sig → Signature of report tarball
2. ubi8-8.3-reports-signature.tar.gz
   • Scan results and justifications
   • documentation.json
     • "image": "registry1.dso.mil/ironbank/redhat/ubi/ubi8@sha256:a5e5632ee49c9b9a7ff9be8ae4b0cd1ed0f2e568826e99eea67cf198e5e7eff8"
3. Docker manifest's sha256 matches the image digest sha256
   • Multiple layers listed in manifest JSON with sha256 hashes
4. Each docker layer tar matches their sha256 Docker manifest JSON entry

ubi8-8.3-reports-signature.sig (outside the archive, detached signature)

PGP signature of reports archive (Maybe an x509 signature if we get an appropriate key to use instead)

ubi8-8.3-reports-signature.tar.gz

reports (name may change in the future)
├── LICENSE
├── README.md
├── anchore
│   ├── anchore-version.txt
│   ├── anchore_api_gates_full.json
│   ├── anchore_gates.json
│   └── anchore_security.json
├── csvs
│   ├── all_scans.xlsx (deprecated)
│   ├── anchore_gates.csv
│   ├── anchore_security.csv
│   ├── oscap.csv
│   ├── oval.csv
│   ├── summary.csv
│   ├── tl.csv
│   └── ubi8:8.3-120979-justifications.xlsx (deprecated)
├── documentation.json
├── manifest.json (deprecated)
├── openscap
│   ├── oscap-version.txt
│   ├── report-cve.html
│   ├── report-cve.xml
│   └── report.html
├── signature.sig (deprecated)
├── twistlock
│   ├── twistlock-version.txt
│   └── twistlock_cve.json
├── ubi8-120979.sig (deprecated)
├── ubi8-120979.tar (deprecated)
└── ubi8-8.3.tar (deprecated)

documentation.json

TODO: jsonschema

{
  "image": "registry1.dso.mil/ironbank/redhat/ubi/ubi8@sha256:a5e5632ee49c9b9a7ff9be8ae4b0cd1ed0f2e568826e99eea67cf198e5e7eff8",
  "timestamp": "2021-01-06T18:34:29.097Z",
  "git": {
    "hash": "47d385a451a6711f09a720b74955284188becc9c",
    "branch": "master"
  },
  "tools": {
    "anchore": {
      "version": "2.4.1"
    },
    "twistlock": {
      "version": "2.4.1"
    },
    "openSCAP": {
      "version": "0.1.53"
    }
  }
}

anchore_gates.csv header

image_id,repo_tag,trigger_id,gate,trigger,check_output,gate_action,policy_id,matched_rule_id,whitelist_id,whitelist_name,inherited,Justification

anchore_security.csv header

tag,cve,severity,package,package_path,fix,url,inherited,Justification

oscap.csv header

title,ruleid,result,severity,identifiers,refs,desc,rationale,scanned_date,Justification

oval.csv header

id,result,cls,ref,title

summary.csv header

Scan,Automated Findings,Manual Checks,Total

tl.csv header

id,cvss,desc,link,packageName,packageVersion,severity,status,vecStr,Justification