Improvements to docker tag artifact caching fixes
Description
There were a couple issues in the original docker-tag-sha-validation MR (!1470 (closed)) that this MR addresses, to be merged back into that branch.
- If a container artifact cache miss took place, the cache was cleared but the artifact was not then redownloaded. This would result in an immediate failure. This resolves that by switching the
elifchange back toif. - For container artifacts,
delete_checksumwas changed todelete_manifestandvalidate_checksumwas changed tovalidate_manifest. This necessitated artifact type checks in various places in the code to call the right method depending on if the artifact was a container artifact or file artifact. In one place, the oldAbstractContainerArtifact.delete_checksummethod was still being called even though it didn't exist anymore. I've simplified all this by adding a commonvalidate_cachemethod anddelete_artifact_and_cachemethod toAbstractArtifact, so we no longer need artifact type checks in various places to know which method to call for cache validation and for artifact / cache deletion. -
self.authfilewas being passed into Skopeo, butself.authfiledid not exist. I added it to theAbstractContainerArtifactclass to mirror how its set on theContainerArtifactclass. - I removed some of the debug printing to print out files that didn't seem to be used anywhere, just to clean things up.
- I cleaned up the code_checker to remove the exclusion of
downloader.py.
Risk
This is being merged into the original docker-tag-sha-validation MR so merging this is low risk. Once that MR merges though, the risk to the pipeline is artifacts being cached incorrectly somehow.
Rollback Plan
We can revert the MR.
Testing
cache hit, pulling an ubuntu:20.04 image artifact https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/594619
rebuild ubuntu, cache miss https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595042
run again, cache hit https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595055
add a file artifact to hardening manifest resources, file downloads https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595094
re-run, cache hit https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595105
file modified, cache miss https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595116
re-run again, cache hit https://code-ib-mario.staging.dso.mil/dsop/c.pawlukowsky/harbor-core/-/jobs/595127