K8s-monitoring for Alloy to move from v1 to v2

added statusready-to-work teamObservability labels

added alloy label

This relates to #70 (closed). Closing this story will close issue #70 (closed).

changed iteration to Big Bang Iterations Jan 21, 2025 - Feb 3, 2025

mentioned in issue #70 (closed)

added priority2 label

changed the description

marked this issue as blocking #72 (closed)

assigned to @kliu

assigned to @blairbowden

Need to work through security contexts Warning InstallFailed 2m41s helm-controller Helm install failed for release monitoring/monitoring-alloy with chart k8s-monitoring@2.0.4-bb.0: failed to create resource │ │ : admission webhook "validate.kyverno.svc-fail" denied the request: │ │ │ │ resource DaemonSet/monitoring/monitoring-alloy-alloy-receiver was blocked due to the following policies │ │ │ │ restrict-capabilities: │ │ autogen-capabilities: 'validation failure: validation error: Adding of additional │ │ capabilities beyond the allowed set is not allowed. The fields spec.containers[*].securityContext.capabilities.add, │ │ spec.initContainers[*].securityContext.capabilities.add, and spec.ephemeralContainers[*].securityContext.capabilities.add │ │ must be in the allowed list. rule autogen-capabilities failed at path /securityContext/capabilities/add/' │ │ │ │ Last Helm logs: │ │ │ │ 2025-01-30T18:22:04.316363188Z: Created a new ClusterRole called "monitoring-alloy-alloy-receiver" in │ │ │ │ 2025-01-30T18:22:04.328821525Z: Created a new ClusterRole called "monitoring-alloy-kepler-clusterrole" in │ │ │ │ 2025-01-30T18:22:04.340925507Z: Created a new ClusterRoleBinding called "monitoring-alloy-alloy-metrics" in │ │ │ │ 2025-01-30T18:22:04.353482965Z: Created a new ClusterRoleBinding called "monitoring-alloy-alloy-receiver" in │ │ │ │ 2025-01-30T18:22:04.365104041Z: Created a new ClusterRoleBinding called "monitoring-alloy-kepler-clusterrole-binding" in │ │ │ │ 2025-01-30T18:22:04.38368784Z: Created a new Service called "monitoring-alloy-alloy-metrics-cluster" in monitoring │ │ │ │ 2025-01-30T18:22:04.426279471Z: Created a new Service called "monitoring-alloy-alloy-metrics" in monitoring │ │ │ │ 2025-01-30T18:22:04.472364663Z: Created a new Service called "monitoring-alloy-alloy-receiver" in monitoring │ │ │ │ 2025-01-30T18:22:04.537053454Z: Created a new Service called "monitoring-alloy-kepler" in monitoring │ │ │ │ 2025-01-30T18:22:04.606001954Z: Created a new Service called "monitoring-alloy-windows-exporter" in monitoring │ │ Normal UninstallSucceeded 2m31s helm-controller Helm uninstall remediation for release monitoring/monitoring-alloy.v1 with chart k8s-monitoring@2.0.4-bb.0 succeeded │ │ Warning InstallFailed 2m9s helm-controller Helm install failed for release monitoring/monitoring-alloy with chart k8s-monitoring@2.0.4-bb.0: 3 errors occurred: │ │ * admission webhook "validate.kyverno.svc-fail" denied the request: │ │ │ │ resource DaemonSet/monitoring/monitoring-alloy-alloy-receiver was blocked due to the following policies │ │ │ │ restrict-capabilities: │ │ autogen-capabilities: 'validation failure: validation error: Adding of additional │ │ capabilities beyond the allowed set is not allowed. The fields spec.containers[*].securityContext.capabilities.add, │ │ spec.initContainers[*].securityContext.capabilities.add, and spec.ephemeralContainers[*].securityContext.capabilities.add │ │ must be in the allowed list. rule autogen-capabilities failed at path /securityContext/capabilities/add/' │ │ │ │ * admission webhook "validate.kyverno.svc-fail" denied the request: │ │ │ │ resource DaemonSet/monitoring/monitoring-alloy-kepler was blocked due to the following policies │ │ │ │ restrict-host-ports: │ │ autogen-host-ports: 'validation failure: validation error: The host port used is │ │ not allowed. The fields spec.containers[*].ports[*].hostPort, spec.initContainers[*].ports[*].hostPort, │ │ and spec.ephemeralContainers[*].ports[*].hostPort must only include ports from │ │ the allowed list. rule autogen-host-ports failed at path /ports/0/hostPort/' │ │ restrict-image-registries: │ │ autogen-validate-registries: 'validation failure: validation error: Image registry │ │ is not in the approved list. rule autogen-validate-registries failed at path /image/' │ │ │ │ * admission webhook "validate.kyverno.svc-fail" denied the request: │ │ │ │ resource StatefulSet/monitoring/monitoring-alloy-alloy-metrics was blocked due to the following policies │ │ │ │ restrict-capabilities: │ │ autogen-capabilities: 'validation failure: validation error: Adding of additional │ │ capabilities beyond the allowed set is not allowed. The fields spec.containers[*].securityContext.capabilities.add, │ │ spec.initContainers[*].securityContext.capabilities.add, and spec.ephemeralContainers[*].securityContext.capabilities.add │ │ must be in the allowed list. rule autogen-capabilities failed at path /securityContext/capabilities/add/' │ │

Disabling clusterMetrics until:

windows-exporter
kepler
opencost

has been added to iron bank

alloy-receiver is getting the following error

│ Error: /etc/alloy/config.alloy:226:29: illegal character U+003A ':'                                                                                                                                                                                                                                 │
│                                                                                                                                                                                                                                                                                                     │
│ Error: /etc/alloy/config.alloy:226:29: expected block label, got ILLEGAL                                                                                                                                                                                                                            │
│ Error: could not perform the initial load successfully                                                                                                                                                                                                                                              │
│ interrupt received                                                                                                                                                                                                                                                                                  │
│ Stream closed EOF for monitoring/monitoring-alloy-alloy-receiver-5mk6p (alloy)

Right now with recent commits pods are running except for metrics. disabling the netpols, the metric pod was able to run would indicate a netpol needs to be created.