UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects

Draft: Resolve "Provide tooling to migrate an active cluster from Operator Istio to Helm Istio."

Open Draft: Resolve "Provide tooling to migrate an active cluster from Operator Istio to Helm Istio."
2283-provide-tooling-to-migrate-an-active-cluster-from-operator-istio-to-helm-istio into master
3 unresolved threads
Open Greg M requested to merge 2283-provide-tooling-to-migrate-an-active-cluster-from-operator-istio-to-helm-istio into master
3 unresolved threads
Compare
and
Show latest version
1 file
+ 29
− 27
Compare changes
  • Side-by-side
  • Inline
docs/guides/using-bigbang/migrating-istio-in-bb3.0.md
+ 29
− 27
@@ -2,29 +2,31 @@
### *The new Istio Helm packages are BETA in Big Bang 2.x and will be stable in 3.0*
### Step 1 : Remove Istio from your deployment
Before upgrading to version 3.0 of Big Bang with the new Helm-based Istio packages, we first need to disable Istio and Istio's Operator packages in our 2.x deployment of Big Bang. We do this by disabling the two obsolete [Istio packages in our Gitops configuration](https://repo1.dso.mil/kipten/template/-/blob/main/package-strategy/configmap.yaml?ref_type=heads#L18-22).
### Step 1 : Remove Istio from your current deployment
Before upgrading to the new helm-based Istio packages, first disable the istio and istioOperator packages:
```yaml
istio:
enabled: false
istioOperator:
enabled: false
```
After a few minutes, all pods in both the `istio-system` and `istio-operator` namespaces should have terminated. However, due due to Istio's finalizer, it's likely that the `istio-system` namespace will be stuck in the `terminating` state. We can force the deletion of this namespace with the following:
After a few minutes, all pods in both the `istio-system` and `istio-operator` namespaces will have terminated. However, due to Istio's finalizer, the `istio-system` namespace will be stuck in the _**terminating**_ state.
Force the deletion of this namespace:
```bash
kubectl get ns istio-system -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/istio-system/finalize" -f -
```
Both namespaces are now removed yet other remnants of Istio still linger in the cluster including custom resources. These also need to be removed as they will be re-instantiated via the helm deployment of Istio. There are various methods to accomplish this feat, but by far the easiest way to do this is by using the [istioctl CLI tool](https://istio.io/latest/docs/ops/diagnostic-tools/istioctl/).
Both Istio namespaces are now removed yet other remnants still linger, not limited to, but including custom resources. Remove them as they will be recreated via the helm deployment of Istio. The quickest way to do this is by using the [istioctl CLI tool](https://istio.io/latest/docs/ops/diagnostic-tools/istioctl/).
If you're on Mac or Linux, you can quickly install it with:
On macOS or Linux install it with:
```bash
brew install istioctl
```
To complete the removal of the Istio remnants, purge [as per Istio's documentation](https://istio.io/latest/docs/setup/install/istioctl/#uninstall-istio):
To complete the removal of remaining Istio components, purge as per [Istio's documentation](https://istio.io/latest/docs/setup/install/istioctl/#uninstall-istio):
```bash
istioctl uninstall --purge
```
Accept the prompt and these remnants are removed:
Accept the prompt to proceed:
```bash
All Istio resources will be pruned from the cluster
Proceed? (y/N) y
@@ -55,27 +57,29 @@ Proceed? (y/N) y
✔ Uninstall complete
```
### Step 2 : Deploy the new Helm based version of Istio
Enabling the Helm based version of Istio entails enabling the `istioCore` package that provides both the `istio-base` and `istiod` charts. The `istioGatewayPublic` package provides the default ingress gateway for most packages and the `istioGatewayPassthrough` provides a secondary non-TLS gated gateway for specific apps that require this like Keycloak.
### Step 2 : Deploy the new Helm Istio package
Enabling the Helm based version of Istio entails enabling the `istioCore` package that provides both the `istio-base` and `istiod` charts. The `istioGateway` package provides the ability to add one or more egress gateways:
```yaml
istioCore:
enabled: true
istioGatewayPublic:
enabled: true
istioGatewayPassthrough:
istioGateway:
enabled: true
```
You can check that new gateway recieves an external IP (from MetalLB or AWS LB) with:
When migrating gateway configurations from Operator to Operatorless, see [the examples here](https://repo1.dso.mil/big-bang/bigbang/-/blob/master/chart/values.yaml#L209-263) as a reference to format values.
After deployment, check that new gateway recieves an external IP (from MetalLB or AWS LB) with:
```bash
kubectl get svc -n istio-gateway
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
public-ingressgateway LoadBalancer 10.43.110.109 172.16.88.88 15021:31155/TCP,80:31302/TCP,443:31046/TCP
```
You'll notice at this point that services are unreachable with errors like:
Notice that services are now unreachable with errors like:
```
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
```
The reason for this is becuase we now need to cycle istio-injected pods so that they connect to the new service mesh. The simple bash script below will iterate through all istio-injected namespaces and cycles all pods therein:
To resolve this issue, cycle all Istio injected pods allowing their reconnection to the new service mesh.
This simple bash script will iterate through all `istio-injected` namespaces and recycle pods:
```bash
# in istio-injected namespaces, recycle pods
for namespace in `kubectl get ns -o custom-columns=:.metadata.name --no-headers -l istio-injection=enabled`
@@ -88,13 +92,13 @@ do
done
```
### Optionally reconcile Helm Releases
You may also need to initiate a reconciliation to all of Big Bang's helm releases managed by `flux`.
It may be necessary, but not likely, to synchronize helm releases managed by Flux. Typically, this can occur when a Gitops deployment of Big Bang sees its helm resources get out of sync during an upgrade.
This step does require that `flux` is [installed locally](https://fluxcd.io/flux/installation/). Linux and Mac users can simply:
The `flux` CLI must be [installed locally](https://fluxcd.io/flux/installation/) -- on macOS and Linux:
```bash
brew install fluxcd/tap/flux
```
This simple bash script will iterate through all of Big Bang managed Helm release and prompt `flux` to [reconcile](https://fluxcd.io/flux/cmd/flux_reconcile_helmrelease) each HR one at a time waiting for them to complete. Typically, this can be useful when managing a Gitops deployment of Big Bang during upgrades or when helm and its dependencies get out of sync.
This bash script iterates through all helm releases managed by Big Bang and has `flux` initiate a [reconciliation](https://fluxcd.io/flux/cmd/flux_reconcile_helmrelease) on each HR one by one:
```bash
# reconcile all of big bang's helm releases w/ flux
for hr in `kubectl get hr --no-headers -n bigbang | awk '{ print $1 }'`
@@ -103,16 +107,14 @@ do
flux reconcile hr $hr -n bigbang --with-source
done
```
### Enjoy your new helm based deploymenbt of Istio!
At this point all services in the cluster should be reachable via the new service mesh.
# Other Notes
At this point all services in the cluster should be reachable via the new service mesh.
## Other Notes
- The Istio Operator has reached its end of life not supporting versions of Istio after 1.23
- The Istio Operator has reached its end of life and does not support versions of Istio after 1.23
- An LTS release, Istio 1.23 is only supported [through May 2025](https://istio.io/latest/docs/releases/supported-releases/#:~:text=1.25%2C%201.26%2C%201.27-,1.23,-Yes)
- The migration from Operator to Helm maintains a consistent version 1.23 to reduce the complexity of the process
- In order to continue utilizing Istio in Big Bang releases >=3.0 this migration is required
- An upgrade to version 1.24 of Istio will soon follow in version 3.1 or version 3.2 of Big Bang in mid-2025
- A rollback from Helm Istio to Operator Istio is possible by reversing the migration steps process
- In order to continue utilizing Istio in Big Bang releases beyond 3.0, this migration is required
- An upgrade to version 1.25 of Istio will soon follow in version 3.1 or version 3.2 of Big Bang in mid-2025
- A rollback from Helm Istio to Operator Istio is possible by reversing the migration process above
- [Diagnostic Tools for Istio](https://istio.io/latest/docs/ops/diagnostic-tools) and [Troubleshooting tips](https://github.com/istio/istio/wiki/Troubleshooting-Istio) can be of assistance for troubled migrations
- Similarly, there is [an Istio manifest tool](https://github.com/istio/istio/pull/52281) that can be used to compare pre and post upgrades
\ No newline at end of file

UNCLASSIFIED - NO CUI