Update Istio to 1.9
All threads resolved!
All threads resolved!
Package Owner Merge Request
Package Changes
Istio control plane and Istio operator update to upstream Istio 1.9.7.
MRs associated with this change.
- https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-controlplane/-/merge_requests/60
- https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-operator/-/merge_requests/18
- https://repo1.dso.mil/platform-one/big-bang/apps/core/authservice/-/merge_requests/38
Additional Details
Edited by Michael McLeroy
Merge request reports
Activity
changed milestone to %1.14.0
added 1 commit
- 08c519a6 - chore(istio-oper): update pull secret to new format
added istio label
added authservice label
Resolved by Michael McLeroyI believe the following items are still required:
- Canary upgrade documentation
- A new IB image for https://repo1.dso.mil/dsop/opensource/istio/1.9/proxyv2/-/issues/7
- Last reply by Michael McLeroy
Thank you @michaelmcleroy
dsop/opensource/istio/1.9/proxyv2#7 has been resolved now.
We are working on documentation of manual upgrade following the upstream documentation: https://istio.io/latest/docs/setup/upgrade/canary/. We ran into an issue during the upgrade and if we can resolve it then we can have the documentation ready on Monday.
mentioned in merge request !737 (merged)
- Resolved by Michael McLeroy
@cdevarenne It looks like the upgrade pipeline is failing. This would not be doing a Canary upgrade, but just an in-place upgrade. Can you take a look and see what is going on?
I restarted the pipeline just now to get a 2nd run on it so you'd have 2 data points.
- Last reply by Michael McLeroy
added 61 commits
-
08c519a6...32abc006 - 60 commits from branch
master - f2abdb24 - Merge branch 'master' into 628-update-istio-to-1-9
-
08c519a6...32abc006 - 60 commits from branch
added statusreview label
- Resolved by Michael McLeroy
@cdevarenne During an upgrade of authservice to 1.9.7, the
pod/authservice-authservice-redis-replicas-*have problems upgrading and restart. After failing 3-5 times they eventually work (takes about 5-7 minutes). Here is the error they are throwing:redis 1:S 11 Aug 2021 19:30:29.651 * MASTER <-> REPLICA sync started redis 1:S 11 Aug 2021 19:30:29.651 # Error condition on socket for SYNC: Connection refused redis 1:S 11 Aug 2021 19:30:30.653 * Connecting to MASTER authservice-authservice-redis-master-0.authservice-authservice-redis-headless.authservice.svc.cluster.local:6379I believe we have 2 replicas running by default.
Can you investigate if this is expected behavior or a side effect of the EnvoyFilters in authservice since those switch when the new istio is deployed?
added statusdoing label and removed statusreview label
@michaelmcleroy I'm able to reproduce the issue. looking into it.
my findings so far:
k get pods -n authservice -w NAME READY STATUS RESTARTS AGE authservice-5495b55b87-wbwvx 2/2 Running 0 25m authservice-haproxy-sso-5ff7d67548-2clp2 2/2 Running 0 21m *** upgrade to 1.9.7*** redis-clean-upgrade-nmt7r 0/1 ContainerCreating 0 3s redis-clean-upgrade-nmt7r 1/1 Running 0 12s redis-clean-upgrade-nmt7r 0/1 Completed 0 32s authservice-fddd9664b-dczps 0/2 Pending 0 1s authservice-fddd9664b-dczps 0/2 Pending 0 1s authservice-fddd9664b-dczps 0/2 Init:0/1 0 1s authservice-authservice-redis-master-0 0/2 Pending 0 1s authservice-authservice-redis-replicas-0 0/2 Pending 0 1s authservice-authservice-redis-master-0 0/2 Pending 0 4s authservice-authservice-redis-replicas-0 0/2 Pending 0 4s authservice-authservice-redis-master-0 0/2 Init:0/1 0 4s authservice-authservice-redis-replicas-0 0/2 Init:0/1 0 4s authservice-authservice-redis-master-0 0/2 PodInitializing 0 18s authservice-fddd9664b-dczps 0/2 PodInitializing 0 19s authservice-authservice-redis-replicas-0 0/2 PodInitializing 0 19s authservice-fddd9664b-dczps 0/2 Running 0 32s authservice-authservice-redis-master-0 0/2 Running 0 36s authservice-authservice-redis-replicas-0 0/2 Running 0 38s authservice-fddd9664b-dczps 1/2 Running 0 39s authservice-fddd9664b-dczps 2/2 Running 0 41s authservice-5495b55b87-wbwvx 2/2 Terminating 0 27m authservice-authservice-redis-master-0 1/2 Running 0 45s authservice-authservice-redis-replicas-0 0/2 Running 1 72s authservice-5495b55b87-wbwvx 0/2 Terminating 0 28m authservice-5495b55b87-wbwvx 0/2 Terminating 0 28m authservice-5495b55b87-wbwvx 0/2 Terminating 0 28m authservice-authservice-redis-master-0 2/2 Running 0 88s authservice-authservice-redis-replicas-0 0/2 Running 2 108s authservice-authservice-redis-replicas-0 1/2 Running 2 108s authservice-authservice-redis-replicas-0 2/2 Running 2 114s authservice-authservice-redis-replicas-1 0/2 Pending 0 0s authservice-authservice-redis-replicas-1 0/2 Pending 0 3s authservice-authservice-redis-replicas-1 0/2 Init:0/1 0 3s authservice-authservice-redis-replicas-1 0/2 PodInitializing 0 6s authservice-authservice-redis-replicas-1 0/2 Running 0 9s authservice-authservice-redis-replicas-1 0/2 Running 1 36s authservice-authservice-redis-replicas-1 1/2 Running 1 49s authservice-authservice-redis-replicas-1 2/2 Running 1 53s redis-clean-upgrade-nmt7r 0/1 Terminating 0 4m22s redis-clean-upgrade-nmt7r 0/1 Terminating 0 4m22s redis-clean-upgrade-7mt4t 0/1 Pending 0 0s redis-clean-upgrade-7mt4t 0/1 Pending 0 0s redis-clean-upgrade-7mt4t 0/1 ContainerCreating 0 0s redis-clean-upgrade-7mt4t 1/1 Running 0 1s redis-clean-upgrade-7mt4t 0/1 Completed 0 21s authservice-authservice-redis-master-0 2/2 Terminating 0 3m14s authservice-authservice-redis-replicas-1 2/2 Terminating 1 79s authservice-authservice-redis-master-0 0/2 Terminating 0 3m20s authservice-authservice-redis-replicas-1 0/2 Terminating 1 85s authservice-authservice-redis-replicas-1 0/2 Terminating 1 90s authservice-authservice-redis-replicas-1 0/2 Terminating 1 90s authservice-authservice-redis-master-0 0/2 Terminating 0 3m25s authservice-authservice-redis-master-0 0/2 Terminating 0 3m25s authservice-authservice-redis-replicas-1 0/2 Pending 0 1s authservice-authservice-redis-replicas-1 0/2 Pending 0 1s authservice-authservice-redis-master-0 0/2 Pending 0 1s authservice-authservice-redis-master-0 0/2 Pending 0 1s authservice-authservice-redis-replicas-1 0/2 Init:0/1 0 1s authservice-authservice-redis-master-0 0/2 Init:0/1 0 1s authservice-authservice-redis-replicas-1 0/2 PodInitializing 0 4s authservice-authservice-redis-master-0 0/2 PodInitializing 0 5s authservice-authservice-redis-replicas-1 0/2 Running 0 7s authservice-authservice-redis-master-0 0/2 Running 0 8s authservice-authservice-redis-replicas-0 1/2 Running 2 3m34s authservice-authservice-redis-master-0 1/2 Running 0 11s authservice-authservice-redis-replicas-0 1/2 Running 3 3m37s authservice-authservice-redis-replicas-1 1/2 Running 0 19s authservice-authservice-redis-master-0 2/2 Running 0 19s authservice-authservice-redis-replicas-1 2/2 Running 0 21s authservice-authservice-redis-replicas-0 2/2 Running 3 3m54s authservice-authservice-redis-replicas-0 2/2 Terminating 3 3m54s authservice-authservice-redis-replicas-0 0/2 Terminating 3 4m authservice-authservice-redis-replicas-0 0/2 Terminating 3 4m8s authservice-authservice-redis-replicas-0 0/2 Terminating 3 4m8s authservice-authservice-redis-replicas-0 0/2 Pending 0 0s authservice-authservice-redis-replicas-0 0/2 Pending 0 0s authservice-authservice-redis-replicas-0 0/2 Init:0/1 0 0s authservice-authservice-redis-replicas-0 0/2 Init:0/1 0 3s authservice-authservice-redis-replicas-0 0/2 PodInitializing 0 4s authservice-authservice-redis-replicas-0 0/2 Running 0 7s authservice-authservice-redis-replicas-0 0/2 Running 1 32s authservice-authservice-redis-replicas-0 1/2 Running 1 52s authservice-authservice-redis-replicas-0 2/2 Running 1 55sfor some reason redis-clean-upgrade runs twice.
Edited by Stanislav Bondarenko@michaelmcleroy I was able to verify that it's not related to envoyfilter changes. I'm seeing similar behavior with 0.4.0-bb.10
$ kubectl get pods -n authservice -w NAME READY STATUS RESTARTS AGE authservice-5495b55b87-bfkhj 2/2 Running 0 10m authservice-haproxy-sso-5ff7d67548-dkdw5 2/2 Running 0 6m9s redis-clean-upgrade-zdpl9 0/1 Completed 0 97s authservice-authservice-redis-replicas-0 0/2 Running 1 66s authservice-authservice-redis-master-0 2/2 Running 0 67s authservice-5b5f58579d-877gs 1/2 Running 0 67s authservice-5b5f58579d-877gs 2/2 Running 0 71s authservice-5495b55b87-bfkhj 2/2 Terminating 0 10m authservice-5b5f58579d-877gs 1/2 Running 1 72s authservice-5b5f58579d-877gs 2/2 Running 1 72s authservice-authservice-redis-replicas-0 1/2 Running 1 79s authservice-authservice-redis-replicas-0 2/2 Running 1 81s authservice-authservice-redis-replicas-1 0/2 Pending 0 0s authservice-authservice-redis-replicas-1 0/2 Pending 0 5s authservice-authservice-redis-replicas-1 0/2 Init:0/1 0 5s authservice-authservice-redis-replicas-1 0/2 PodInitializing 0 8s authservice-authservice-redis-replicas-1 0/2 Running 0 11s authservice-5495b55b87-bfkhj 1/2 Terminating 0 10m authservice-5495b55b87-bfkhj 0/2 Terminating 0 10m authservice-5495b55b87-bfkhj 0/2 Terminating 0 10m authservice-5495b55b87-bfkhj 0/2 Terminating 0 10m authservice-authservice-redis-replicas-1 0/2 Running 1 37s authservice-authservice-redis-replicas-1 1/2 Running 1 55s authservice-authservice-redis-replicas-1 1/2 Running 2 67s authservice-authservice-redis-replicas-1 2/2 Running 2 86s redis-clean-upgrade-zdpl9 0/1 Terminating 0 3m25s redis-clean-upgrade-zdpl9 0/1 Terminating 0 3m25s redis-clean-upgrade-zbs2h 0/1 Pending 0 0s redis-clean-upgrade-zbs2h 0/1 Pending 0 0s redis-clean-upgrade-zbs2h 0/1 ContainerCreating 0 0s redis-clean-upgrade-zbs2h 1/1 Running 0 3s redis-clean-upgrade-zbs2h 0/1 Completed 0 23s authservice-authservice-redis-master-0 2/2 Terminating 0 3m22s authservice-authservice-redis-replicas-1 2/2 Terminating 2 2m authservice-authservice-redis-replicas-1 0/2 Terminating 2 2m8s authservice-authservice-redis-master-0 0/2 Terminating 0 3m30s authservice-authservice-redis-replicas-1 0/2 Terminating 2 2m18s authservice-authservice-redis-replicas-1 0/2 Terminating 2 2m18s authservice-authservice-redis-master-0 0/2 Terminating 0 3m40s authservice-authservice-redis-master-0 0/2 Terminating 0 3m40s authservice-authservice-redis-replicas-1 0/2 Pending 0 1s authservice-authservice-redis-replicas-1 0/2 Pending 0 1s authservice-authservice-redis-master-0 0/2 Pending 0 2s authservice-authservice-redis-replicas-1 0/2 Init:0/1 0 2s authservice-authservice-redis-master-0 0/2 Pending 0 2s authservice-authservice-redis-master-0 0/2 Init:0/1 0 3s authservice-authservice-redis-replicas-0 1/2 Running 1 3m45s authservice-authservice-redis-master-0 0/2 PodInitializing 0 7s authservice-authservice-redis-replicas-1 0/2 PodInitializing 0 7s authservice-authservice-redis-replicas-0 1/2 Running 2 3m48s authservice-authservice-redis-replicas-1 0/2 Running 0 10s authservice-authservice-redis-master-0 0/2 Running 0 10s authservice-authservice-redis-master-0 1/2 Running 0 17s authservice-authservice-redis-replicas-0 1/2 Running 3 4m17s authservice-authservice-redis-replicas-1 0/2 Running 1 38s authservice-authservice-redis-replicas-1 1/2 Running 1 53s authservice-authservice-redis-replicas-0 2/2 Running 3 4m46s authservice-authservice-redis-replicas-1 2/2 Running 1 67s authservice-authservice-redis-replicas-0 2/2 Terminating 3 4m46s authservice-authservice-redis-master-0 2/2 Running 0 68s authservice-authservice-redis-replicas-0 0/2 Terminating 3 4m52s authservice-authservice-redis-replicas-0 0/2 Terminating 3 5m1s authservice-authservice-redis-replicas-0 0/2 Terminating 3 5m1s authservice-authservice-redis-replicas-0 0/2 Pending 0 1s authservice-authservice-redis-replicas-0 0/2 Pending 0 1s authservice-authservice-redis-replicas-0 0/2 Init:0/1 0 1s authservice-authservice-redis-replicas-0 0/2 Init:0/1 0 3s authservice-authservice-redis-replicas-0 0/2 PodInitializing 0 4s authservice-authservice-redis-replicas-0 0/2 Running 0 7s authservice-authservice-redis-replicas-0 0/2 Running 1 32s authservice-authservice-redis-replicas-0 0/2 Running 2 64s authservice-authservice-redis-replicas-0 1/2 Running 2 67s authservice-authservice-redis-replicas-0 2/2 Running 2 71s $ kubectl get gitrepositories -A NAMESPACE NAME URL READY STATUS AGE bigbang environment-repo https://github.com/sbko/bigbang-template.git True Fetched revision: testing/5e2427c63abf2be4eeeb67d22d9363b8b65552fe 23m bigbang bigbang https://repo1.dso.mil/platform-one/big-bang/bigbang.git True Fetched revision: 1.13.0/eebd31255dc85a005dcdfbb1c70e4e50f879da61 22m bigbang authservice https://repo1.dso.mil/platform-one/big-bang/apps/core/authservice.git True Fetched revision: 0.4.0-bb.10/dac52eb5a7bffefc908c4fb8ebb0da6df885b960 22m bigbang haproxy https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/haproxy True Fetched revision: 1.1.2-bb.0/7e4d75c26c06e60f9bfe753a73ac0dbba73c0d47 22m bigbang istio-operator https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-operator.git True Fetched revision: 1.8.4-bb.2/8734a3ca64de5e4ac787c19e0861f525de6c1355 22m bigbang gatekeeper https://repo1.dso.mil/platform-one/big-bang/apps/core/policy.git True Fetched revision: 3.5.1-bb.4/cc040ca14cce36a1b2da8b39c492048e4b66d4c5 22m bigbang fluentbit https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit.git True Fetched revision: 0.15.15-bb.0/1ef75a508c1c28dc2eedbff7b185535de9fe0e52 22m bigbang cluster-auditor https://repo1.dso.mil/platform-one/big-bang/apps/core/cluster-auditor.git True Fetched revision: 0.3.0-bb.4/3c5138e8b1d15ce3048a068d87adb3fa8b26b8bb 22m bigbang istio-controlplane https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-controlplane.git True Fetched revision: 1.8.4-bb.6/8abf01fe133b5a8579395816083b8c6df4579a13 22m bigbang jaeger https://repo1.dso.mil/platform-one/big-bang/apps/core/jaeger.git True Fetched revision: 2.23.0-bb.1/97f4154446cf10a03feaeb7ea6de575a19e266db 22m bigbang kiali https://repo1.dso.mil/platform-one/big-bang/apps/core/kiali.git True Fetched revision: 1.36.0-bb.3/af4cef0aaa05bed86929d2806ecb0298c9984299 22m bigbang elasticsearch-kibana https://repo1.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana.git True Fetched revision: 0.1.17-bb.0/33901d1ebfc3839b7448f6daf4e793ffcb0309c6 22m bigbang eck-operator https://repo1.dso.mil/platform-one/big-bang/apps/core/eck-operator.git True Fetched revision: 1.6.0-bb.1/8036c4163e0543559ee8f8a8302ba3bd4ba45eee 22m bigbang monitoring https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring.git True Fetched revision: 14.0.0-bb.3/092174bce9088ba0915239fe96b3adbf9845ebfe 22mI opened an issue - https://repo1.dso.mil/platform-one/big-bang/apps/core/authservice/-/issues/17. I think I know what's the problem.
mentioned in commit 09e1ca8a
changed milestone to %1.15.0
mentioned in issue #632 (closed)
mentioned in issue #633 (closed)
Please register or sign in to reply