Fix bug with ArgoCD applications/project post upgrade hooks
Issue reported from Party Bus. For whoever works this issue. Please communicate any proposed fix in BB <-> PB channel Details ->
Party Bus deploys ArgoCD Applications using the argocd-apps sub chart. In 2023, the upstream ArgoCD helm chart project removed support for creating Applications and Projects from the ArgoCD helm chart values. We added the argocd-apps sub chart as a workaround so that users could continue deploying Applications and Projects as helm values with this chart. When PartyBus deploys Applications this way, they end up being post upgrade hooks, which is the intended solution from the bigbang side. However, this creates a dependency issue between the Application and the Appproj
Essentially the workflow is as follows:
Install the chart, helmrelease reconciles and everything deploys Make a change to the values somewhere, push it, then helmrelease gets stuck for a couple reasons
- Due to the fact that Application and Appproj are post-upgrade hooks, helm sees them and wants to replace them. This is sort of the nature of hooks when we think about stuff like upgrade jobs and such
- The Admin Appproj at this point has been tagged withfinalizer: - resources-finalizer.argocd.argoproj.io. This means that nothing can delete it except for argo. This creates a situation where the HR is completely halted
- The next step we followed was patching the finalizer with null and/or deleting it
- Helmrelease kills the admin appproj and the Application and recreates them, resyncs
- However, steps 3 and 4 are not desirable and we would rather helm not touch these resources after they've been deployed (unless we make changes). We're unsure of the best path forward. While it makes sense to deploy them post CRD install, I don't think we necessarily need to have them in post full install and maybe they're just a regular resource deployment
I believe the issue comes into play when you deploy an Application because now that Application controls the Appproj or maybe thats why it gets labeled with the finalizer stuff and thus becomes un-deletable
The Argo controller goes "Woah, I'm not gonna remove the finalizer label from that appproj because an application attached to the appproj still exists" so Helm just sits there waiting for the controller to delete it... and the controller has said that it won't. But we don't want these things getting deleted anyway.
Proposed solution:
Explore another way to create Applications and Projects that doesn't involve using post upgrade hooks? I'm thinking we define some custom templates in chart/templates/bigbang/ and allow the user to pass in a list of Applications and Projects from the helm values.