UNCLASSIFIED - NO CUI

Improvements to docker tag artifact caching fixes

Description

There were a couple issues in the original docker-tag-sha-validation MR (!1470 (closed)) that this MR addresses, to be merged back into that branch.

  1. If a container artifact cache miss took place, the cache was cleared but the artifact was not then redownloaded. This would result in an immediate failure. This resolves that by switching the elif change back to if.
  2. For container artifacts, delete_checksum was changed to delete_manifest and validate_checksum was changed to validate_manifest. This necessitated artifact type checks in various places in the code to call the right method depending on if the artifact was a container artifact or file artifact. In one place, the old AbstractContainerArtifact.delete_checksum method was still being called even though it didn't exist anymore. I've simplified all this by adding a common validate_cache method and delete_artifact_and_cache method to AbstractArtifact, so we no longer need artifact type checks in various places to know which method to call for cache validation and for artifact / cache deletion.
  3. self.authfile was being passed into Skopeo, but self.authfile did not exist. I added it to the AbstractContainerArtifact class to mirror how its set on the ContainerArtifact class.
  4. I removed some of the debug printing to print out files that didn't seem to be used anywhere, just to clean things up.
  5. I cleaned up the code_checker to remove the exclusion of downloader.py.

Risk

This is being merged into the original docker-tag-sha-validation MR so merging this is low risk. Once that MR merges though, the risk to the pipeline is artifacts being cached incorrectly somehow.

Rollback Plan

We can revert the MR.

Testing

cache hit, pulling an ubuntu:20.04 image artifact https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/594619

rebuild ubuntu, cache miss https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595042

run again, cache hit https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595055

add a file artifact to hardening manifest resources, file downloads https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595094

re-run, cache hit https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595105

file modified, cache miss https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595116

re-run again, cache hit https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595127

Edited by Chris Pawlukowsky

Merge request reports

Loading