Monitoring pods never coming up using the quickstart
Bug
Description
Following the quickstart guide, I'm on Step 11.
I've been running the kubectl get po -A
command multiple times over the past hour, and the monitoring pods go through the pattern of (A) completed, (B) pending, (C) running, then they go back to (A)
BigBang Version
2.4.0
Additional helpful information
I'm on a Windows 11 Home laptop, i7-9750H @ 2.60 GHz, with 16 GB of RAM. I'm running Virtual Box 7.0.8, with an Ubuntu Server 22.04.2 64 bit. The VM has 9 GB of RAM and 4 processors, 128 GB drive.
This is what (A) looks like
vboxuser@bigbang:~/bigbang$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-84bb864455-kkgjv 1/1 Running 0 105m
kube-system coredns-96cc4f57d-mkl24 1/1 Running 0 105m
kube-system metrics-server-ff9dbcb6c-jn2tj 1/1 Running 0 105m
flux-system helm-controller-7dc499fcc7-thqb5 1/1 Running 0 97m
flux-system kustomize-controller-68fd78bd4d-jmlhc 1/1 Running 0 97m
flux-system notification-controller-6466ff5b5b-5bgmp 1/1 Running 0 97m
flux-system source-controller-7bcbcbd976-w8p4g 1/1 Running 0 97m
kyverno kyverno-kyverno-7957494765-zj7kd 1/1 Running 0 92m
kyverno kyverno-kyverno-7957494765-bcvrh 1/1 Running 0 92m
kyverno kyverno-kyverno-7957494765-5ghcm 1/1 Running 0 92m
istio-operator istio-operator-58c4ffc8f-cw9ls 1/1 Running 0 90m
istio-system istiod-569cc7dfd8-slnkc 1/1 Running 0 90m
istio-system public-ingressgateway-77d7b68bb6-zvbm7 1/1 Running 0 89m
monitoring monitoring-monitoring-kube-admission-create--1-42blt 0/3 Completed 0 8s
Then (B) is like this
vboxuser@bigbang:~/bigbang$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-84bb864455-kkgjv 1/1 Running 0 105m
kube-system coredns-96cc4f57d-mkl24 1/1 Running 0 105m
kube-system metrics-server-ff9dbcb6c-jn2tj 1/1 Running 0 105m
flux-system helm-controller-7dc499fcc7-thqb5 1/1 Running 0 97m
flux-system kustomize-controller-68fd78bd4d-jmlhc 1/1 Running 0 97m
flux-system notification-controller-6466ff5b5b-5bgmp 1/1 Running 0 97m
flux-system source-controller-7bcbcbd976-w8p4g 1/1 Running 0 97m
kyverno kyverno-kyverno-7957494765-zj7kd 1/1 Running 0 93m
kyverno kyverno-kyverno-7957494765-bcvrh 1/1 Running 0 93m
kyverno kyverno-kyverno-7957494765-5ghcm 1/1 Running 0 93m
istio-operator istio-operator-58c4ffc8f-cw9ls 1/1 Running 0 90m
istio-system istiod-569cc7dfd8-slnkc 1/1 Running 0 90m
istio-system public-ingressgateway-77d7b68bb6-zvbm7 1/1 Running 0 90m
monitoring monitoring-monitoring-prometheus-node-exporter-b7wwm 0/2 Init:0/1 0 4s
monitoring monitoring-monitoring-kube-operator-7c8dbd86c8-4bdwq 0/2 Init:0/1 0 4s
monitoring monitoring-monitoring-kube-state-metrics-69f9966fd7-x5tk2 0/2 Init:0/1 0 4s
monitoring monitoring-monitoring-grafana-6784bc5754-j58l8 0/4 Pending 0 3s
Then (C) is like this
vboxuser@bigbang:~/bigbang$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-84bb864455-kkgjv 1/1 Running 0 112m
kube-system coredns-96cc4f57d-mkl24 1/1 Running 0 112m
kube-system metrics-server-ff9dbcb6c-jn2tj 1/1 Running 0 112m
flux-system helm-controller-7dc499fcc7-thqb5 1/1 Running 0 105m
flux-system kustomize-controller-68fd78bd4d-jmlhc 1/1 Running 0 105m
flux-system notification-controller-6466ff5b5b-5bgmp 1/1 Running 0 105m
flux-system source-controller-7bcbcbd976-w8p4g 1/1 Running 0 105m
kyverno kyverno-kyverno-7957494765-zj7kd 1/1 Running 0 100m
kyverno kyverno-kyverno-7957494765-bcvrh 1/1 Running 0 100m
kyverno kyverno-kyverno-7957494765-5ghcm 1/1 Running 0 100m
istio-operator istio-operator-58c4ffc8f-cw9ls 1/1 Running 0 98m
istio-system istiod-569cc7dfd8-slnkc 1/1 Running 0 98m
istio-system public-ingressgateway-77d7b68bb6-zvbm7 1/1 Running 0 97m
monitoring monitoring-monitoring-grafana-6784bc5754-j58l8 0/4 Pending 0 7m29s
monitoring monitoring-monitoring-prometheus-node-exporter-b7wwm 2/2 Running 0 7m30s
monitoring monitoring-monitoring-kube-operator-7c8dbd86c8-4bdwq 2/2 Running 0 7m30s
monitoring alertmanager-monitoring-monitoring-kube-alertmanager-0 0/3 Pending 0 7m22s
monitoring monitoring-monitoring-kube-state-metrics-69f9966fd7-x5tk2 2/2 Running 0 7m30s
monitoring prometheus-monitoring-monitoring-kube-prometheus-0 0/3 Pending 0 7m20s
If I keep watching it, I'll see it gets back into a NotReady state
vboxuser@bigbang:~/bigbang$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-84bb864455-kkgjv 1/1 Running 0 48m
kube-system coredns-96cc4f57d-mkl24 1/1 Running 0 48m
kube-system metrics-server-ff9dbcb6c-jn2tj 1/1 Running 0 48m
flux-system helm-controller-7dc499fcc7-thqb5 1/1 Running 0 41m
flux-system kustomize-controller-68fd78bd4d-jmlhc 1/1 Running 0 41m
flux-system notification-controller-6466ff5b5b-5bgmp 1/1 Running 0 41m
flux-system source-controller-7bcbcbd976-w8p4g 1/1 Running 0 41m
kyverno kyverno-kyverno-7957494765-zj7kd 1/1 Running 0 36m
kyverno kyverno-kyverno-7957494765-bcvrh 1/1 Running 0 36m
kyverno kyverno-kyverno-7957494765-5ghcm 1/1 Running 0 36m
istio-operator istio-operator-58c4ffc8f-cw9ls 1/1 Running 0 34m
istio-system istiod-569cc7dfd8-slnkc 1/1 Running 0 33m
istio-system public-ingressgateway-77d7b68bb6-zvbm7 1/1 Running 0 33m
monitoring monitoring-monitoring-kube-admission-create--1-kh7s8 1/3 NotReady 0 11s
Then if it helps, here's what Helm shows
vboxuser@bigbang:~/bigbang$ helm ls -n bigbang
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
bigbang bigbang 1 2023-06-28 13:48:52.704396813 +0000 UTC deployed bigbang-2.4.0
istio-operator-istio-operator istio-operator 1 2023-06-28 13:51:16.615130874 +0000 UTC deployed istio-operator-1.17.2-bb.2 1.17.2
istio-system-istio istio-system 1 2023-06-28 13:51:35.333791842 +0000 UTC deployed istio-1.17.2-bb.2 1.17.2
kyverno-kyverno kyverno 1 2023-06-28 13:49:03.709778254 +0000 UTC deployed kyverno-2.6.5-bb.3 v1.8.5
kyverno-kyverno-policies kyverno 1 2023-06-28 13:50:15.852510374 +0000 UTC deployed kyverno-policies-1.1.0-bb.6 1.1.0