fluentbit fails to spin up in, ci clusters don't have `/etc/machine-id`
need to add /etc/machine-id to gitlab-runner deployment responsible for running in cluster k3s clusters
kubectl describe po -n logging logging-fluent-bit-r6km7
Name: logging-fluent-bit-r6km7
Namespace: logging
Priority: 0
Node: runner-bywhcggb-project-2872-concurrent-0zl2sp/10.42.13.14
Start Time: Thu, 11 Mar 2021 00:09:58 +0000
Labels: app.kubernetes.io/instance=logging-fluent-bit
app.kubernetes.io/name=fluent-bit
controller-revision-hash=57f4cf9dc5
istio.io/rev=default
pod-template-generation=1
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=fluent-bit
service.istio.io/canonical-revision=latest
Annotations: checksum/config: 187fd3e05ebd4ba5f78441b25f155893da764e6af7a9f02ae3635b080a7403b3
prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true
sidecar.istio.io/status:
{"version":"8e6e902b765af607513b28d284940ee1421e9a0d07698741693b2663c7161c11","initContainers":["istio-init"],"containers":["istio-proxy"]...
Status: Pending
IP:
IPs: <none>
Controlled By: DaemonSet/logging-fluent-bit
Init Containers:
istio-init:
Container ID:
Image: registry1.dso.mil/ironbank/opensource/istio-1.7/proxyv2-1.7:1.7.7
Image ID:
Port: <none>
Host Port: <none>
Args:
istio-iptables
-p
15001
-z
15006
-u
1337
-m
REDIRECT
-i
*
-x
-b
*
-d
15090,15021,15020
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 10m
memory: 10Mi
Environment:
DNS_AGENT:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from logging-fluent-bit-token-bkqtr (ro)
Containers:
fluent-bit:
Container ID:
Image: registry1.dso.mil/ironbank/opensource/fluent/fluent-bit:1.6.6
Image ID:
Port: 2020/TCP
Host Port: 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get http://:15020/app-health/fluent-bit/livez delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:15020/app-health/fluent-bit/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
FLUENT_ELASTICSEARCH_PASSWORD: <set to the key 'elastic' in secret 'logging-ek-es-elastic-user'> Optional: false
Mounts:
/etc/machine-id from etcmachineid (ro)
/fluent-bit/etc/custom_parsers.conf from config (rw,path="custom_parsers.conf")
/fluent-bit/etc/fluent-bit.conf from config (rw,path="fluent-bit.conf")
/var/lib/docker/containers from varlibdockercontainers (ro)
/var/log from varlog (rw)
/var/run/secrets/kubernetes.io/serviceaccount from logging-fluent-bit-token-bkqtr (ro)
istio-proxy:
Container ID:
Image: registry1.dso.mil/ironbank/opensource/istio-1.7/proxyv2-1.7:1.7.7
Image ID:
Port: 15090/TCP
Host Port: 0/TCP
Args:
proxy
sidecar
--domain
$(POD_NAMESPACE).svc.cluster.local
--serviceCluster
logging-fluent-bit.logging
--proxyLogLevel=warning
--proxyComponentLogLevel=misc:error
--trust-domain=cluster.local
--concurrency
2
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Readiness: http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
Environment:
JWT_POLICY: third-party-jwt
PILOT_CERT_PROVIDER: istiod
CA_ADDR: istiod.istio-system.svc:15012
POD_NAME: logging-fluent-bit-r6km7 (v1:metadata.name)
POD_NAMESPACE: logging (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
HOST_IP: (v1:status.hostIP)
CANONICAL_SERVICE: (v1:metadata.labels['service.istio.io/canonical-name'])
CANONICAL_REVISION: (v1:metadata.labels['service.istio.io/canonical-revision'])
PROXY_CONFIG: {"proxyMetadata":{"DNS_AGENT":""}}
ISTIO_META_POD_PORTS: [
{"name":"http","containerPort":2020,"protocol":"TCP"}
]
ISTIO_META_APP_CONTAINERS: fluent-bit
ISTIO_META_CLUSTER_ID: Kubernetes
ISTIO_META_INTERCEPTION_MODE: REDIRECT
ISTIO_METAJSON_ANNOTATIONS: {"checksum/config":"187fd3e05ebd4ba5f78441b25f155893da764e6af7a9f02ae3635b080a7403b3"}
ISTIO_META_WORKLOAD_NAME: logging-fluent-bit
ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/logging/daemonsets/logging-fluent-bit
ISTIO_META_MESH_ID: cluster.local
DNS_AGENT:
ISTIO_KUBE_APP_PROBERS: {"/app-health/fluent-bit/livez":{"httpGet":{"path":"/","port":2020,"scheme":"HTTP"},"timeoutSeconds":1},"/app-health/fluent-bit/readyz":{"httpGet":{"path":"/","port":2020,"scheme":"HTTP"},"timeoutSeconds":1}}
Mounts:
/etc/istio/pod from istio-podinfo (rw)
/etc/istio/proxy from istio-envoy (rw)
/var/lib/istio/data from istio-data (rw)
/var/run/secrets/istio from istiod-ca-cert (rw)
/var/run/secrets/kubernetes.io/serviceaccount from logging-fluent-bit-token-bkqtr (ro)
/var/run/secrets/tokens from istio-token (rw)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: logging-fluent-bit
Optional: false
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
varlibdockercontainers:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker/containers
HostPathType:
etcmachineid:
Type: HostPath (bare host directory volume)
Path: /etc/machine-id
HostPathType: File
logging-fluent-bit-token-bkqtr:
Type: Secret (a volume populated by a Secret)
SecretName: logging-fluent-bit-token-bkqtr
Optional: false
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
istio-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-podinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
istio-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 43200
istiod-ca-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-ca-root-cert
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoExecute op=Exists
:NoSchedule op=Exists
node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 43s default-scheduler Successfully assigned logging/logging-fluent-bit-r6km7 to runner-bywhcggb-project-2872-concurrent-0zl2sp
Warning FailedMount 12s (x7 over 44s) kubelet MountVolume.SetUp failed for volume "etcmachineid" : hostPath type check failed: /etc/machine-id is not a file