Reinstalling Istio Controlplane Package Fails

While using BB master, the istio-controlplane package can not be reenabled, i.e. enabled: false/true. To reproduce:

1. Create a k8s cluster.
2. Install BB from master with all pkgs disabled except istio-controlplane and istio-operator. BB, istio, and istio-operator hr's should report Ready=true and everything should work as expected.
3. Disable istio by setting enabled: false for istio and istiooperator in your customer template.
4. Ensure the istio-system ns is deleted. In my experience, the ns does not delete due to the presence of the IstioOperator resource in the istio-system ns. Remove the finalizer for the ns to finish terminating: kubectl patch istiooperator/istiocontrolplane -n istio-system --patch '{"metadata":{"finalizers": []}}' --type=merge
5. Reenable the istio pkgs by setting enabled: true for istio and istiooperator in your customer template. The istio hr fails due to:

$ kubectl get hr/istio -n bigbang -o yaml
...
status:
  conditions:
  - lastTransitionTime: "2021-08-23T21:50:01Z"
    message: install retries exhausted
    reason: InstallFailed
    status: "False"
    type: Ready
  - lastTransitionTime: "2021-08-23T21:50:01Z"
    message: |-
      Helm install failed: Internal error occurred: failed calling webhook "validation.istio.io": Post "https://istiod.istio-system.svc:443/validate?timeout=30s": service "istiod" not found

      Last Helm logs:

      creating 14 resource(s)
    reason: InstallFailed
    status: "False"
    type: Released
  failures: 11
  helmChart: bigbang/bigbang-istio
  installFailures: 4
  lastAttemptedRevision: 1.9.7-bb.0
  lastAttemptedValuesChecksum: 3490f9d7a82fd2e5cff0be9ed760fb06afecf45b
  lastReleaseRevision: 1
  observedGeneration: 2

service "istiod" not found is because the service was not created. The flux helm controller provides no additional details for the cause of the failure:

$ kubectl logs deploy/helm-controller -n flux-system
...
{"level":"info","ts":"2021-08-23T23:27:13.842Z","logger":"controller.helmrelease","msg":"all dependencies are ready, proceeding with release","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"istio","namespace":"bigbang"}
{"level":"info","ts":"2021-08-23T23:27:13.875Z","logger":"controller.helmrelease","msg":"reconcilation finished in 33.035218ms, next run in 1m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"istio","namespace":"bigbang"}
{"level":"error","ts":"2021-08-23T23:27:13.875Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"istio","namespace":"bigbang","error":"install retries exhausted"}
{"level":"info","ts":"2021-08-23T23:27:38.052Z","logger":"controller.helmrelease","msg":"could not find optional Secret 'bigbang/terraform'","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"bigbang","namespace":"bigbang"}
{"level":"info","ts":"2021-08-23T23:27:38.177Z","logger":"controller.helmrelease","msg":"reconcilation finished in 125.842088ms, next run in 1m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"bigbang","namespace":"bigbang"}
{"level":"info","ts":"2021-08-23T23:27:54.835Z","logger":"controller.helmrelease","msg":"all dependencies are ready, proceeding with release","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"istio","namespace":"bigbang"}
{"level":"info","ts":"2021-08-23T23:27:54.866Z","logger":"controller.helmrelease","msg":"reconcilation finished in 30.994814ms, next run in 1m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"istio","namespace":"bigbang"}
{"level":"error","ts":"2021-08-23T23:27:54.867Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"istio","namespace":"bigbang","error":"install retries exhausted"}

The istio operator logs indicate the IstioOperator resource managed by the istio-controlplane pkg does not get created:

$ kubectl logs deploy/istio-operator -n istio-operator
2021-08-23T23:26:03.630020Z	info	leader election cm: istio-operator-lock
2021-08-23T23:26:03.718943Z	info	ControlZ available at 127.0.0.1:9876
2021-08-23T23:26:04.780196Z	info	klog	Throttling request took 1.045697777s, request: GET:https://10.3.240.1:443/apis/autoscaling/v2beta2?timeout=32s
2021-08-23T23:26:04.987448Z	info	Creating operator metrics exporter
2021-08-23T23:26:04.987722Z	info	Registering Components.
2021-08-23T23:26:04.987988Z	info	installer	Adding controller for IstioOperator.
2021-08-23T23:26:04.988264Z	info	installer	Controller added
2021-08-23T23:26:04.988304Z	info	Starting the Cmd.
2021-08-23T23:26:04.988636Z	info	klog	attempting to acquire leader lease istio-operator/istio-operator-lock...
2021-08-23T23:26:05.003153Z	info	klog	successfully acquired lease istio-operator/istio-operator-lock
2021-08-23T23:26:06.111139Z	info	klog	rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
2021-08-23T23:26:06.113046Z	info	klog	rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
2021-08-23T23:26:06.311854Z	info	klog	rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
2021-08-23T23:26:06.313594Z	info	klog	rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
2021-08-23T23:26:06.512645Z	info	klog	admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
2021-08-23T23:26:06.515218Z	info	klog	admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
2021-08-23T23:26:06.613463Z	info	klog	admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
2021-08-23T23:26:06.616811Z	info	klog	admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
2021-08-23T23:26:07.038246Z	info	klog	apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
2021-08-23T23:26:07.326092Z	info	klog	apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition

k8s versions:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.11", GitCommit:"c6a2f08fc4378c5381dd948d9ad9d1080e3e6b33", GitTreeState:"clean", BuildDate:"2021-05-12T12:27:07Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.8-gke.2100", GitCommit:"4cd085fda961821985d176d25b67445c1efb6ba1", GitTreeState:"clean", BuildDate:"2021-07-16T09:22:57Z", GoVersion:"go1.15.13b5", Compiler:"gc", Platform:"linux/amd64"}

@michaelmcleroy PTAL at this issue when you have a moment and let me know a) if you're aware of any issue/caveats or reenabling istio cp pkg and b) if you can repro this issue. This issue is currently blocking my progress with https://repo1.dso.mil/platform-one/big-bang/bigbang/-/issues/587.

changed the description

I found the source of the issue. Although hr's support dependencies for creation, these dependencies are not respected for delete operations. Setting enabled: false for istio and istiooperator causes a race condition where istiooperator resources are deleted before istio resources. A HelmRelease should respect the dependency graph in reverse order for delete operations, xref: https://github.com/fluxcd/flux2/issues/1744.

assigned to @daneyon

added Istio-Support label

added priority8 label

added teamcore/security label

added stale label

removed stale label

removed Istio-Support label

@jasonkrause @micah.nagel @ryan.j.garcia @michaelmcleroy @runyontr (cc @ryan.thompson.44 )

Is this something customers want/need or should the issue be closed?

I think we can close this for the time being...this is reliant on Flux changing behavior unless we want to add a lot of "complex" logic to tear down things via helm post-delete jobs and such...

Agreed, we can document if toggling istio or wanting to full re-install you MUST delete each HelmRelease in addition to the namespaces then everything will be happy.

Not 100% sure but I think this is doable simply by disabling istio (controlplane only) first, giving the operator time to tear everything down, and then disabling the operator. The issue with disabling both at the same time is that they get torn down at the same time and the operator (which would typically handle all the cleanup of the webhook, etc) gets torn down before it can handle deletion of the IstioOperator custom resource.

EDIT: So basically if we just test and document that ^ we might be good. Or worst case document all the things that have to be manually deleted.

added kinddocs label

removed teamcore/security label

added teamIntegration label

removed priority8 label

unassigned @daneyon

added stale label

@daneyon this issue has been inactive for 60 days and is being labelled as marked-for-auto-close. If this issue is still required please take action by removing the stale and marked-for-auto-close labels and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken it will be auto closed in 30 days.

added marked-for-auto-close label

removed stale label

@daneyon this issue has been inactive for 30 days and is being labelled as stale. If this issue is still required please take action by removing the stale label and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken this issue will be auto closed in 60 days.

added stale label

assigned to @kenna81

@kenna81 , does this make sense to include in your docs?

added delete-me label

@daneyon this issue has been closed due to the label delete-me.

closed