Provide RBAC option to run privileged initContainer sysctl

Problem

I am unable to install EK with the N2X flux-bigbang deployment model because of a series of errors.

Environment

• Kubernetes 1.20.6+rke2r1
• AWS
• RKE2

Workaround

My workaround is to deploy EK with helm outside of bigbang directly from this repository as follows:

1. Create logging namespace and bigbang secret
2. Create storage class for EK PVCs
3. Create RBAC resources for privileged sysctl pod
4. Install EK with helm overrides
5. Apply RBAC service account to EK statefulsets with kubectl set serviceaccount sts $RESOURCE $SERVICEACCOUNT

Proposal

Add RBAC resources to EK helm charts and add service account to 2 statefulsets: ek-es-master & ek-es-data.

• ServiceAccount
• ClusterRole
• ClusterRoleBinding
• PodSecurityPolicy (optional, currently created by N2X prior to BigBang installation)

TL;DR

The EK helm charts comment a sysctl initContainer to set vm.max_map_count. The EK installation as-is produces this error:

[1]: initial heap size [1073741824] not equal to maximum heap size [4294967296]; this can cause resize pauses
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Uncommenting the lines to enable the initContainer produces this error:

create Pod logging-ek-es-master-0 in StatefulSet logging-ek-es-master failed error: pods "logging-ek-es-master-0" is forbidden: unable to validate against any pod security policy: [spec.initContainers[1].securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden spec.initContainers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

Until a suitable replacement is available for deprecated PodSecurityPolicies PodSecurityPolicy Deprecation: Past, Present, and Future, I applied the following resources and a permissive PodSecurityPolicy to enable the initContainer.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ek-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ek-psp-role
rules:
- apiGroups:
  - policy
  resourceNames:
  - privileged
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ek-sa-psp-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ek-psp-role
subjects:
- kind: ServiceAccount
  name: ek-sa
  namespace: logging

I performed a helm installation of EK with the following overrides.

elasticsearch:
  imagePullSecrets:
    - name: private-registry

  # Values for master node sets
  master:
    count: 1
    persistence:
      storageClassName: gp2
      size: 5Gi
    resources:
      limits:
        cpu: 1
        memory: 3Gi
    # This privileged initContainer resolves the elasticsearch error:
    # [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
    initContainers:
    # add an init container that adjusts the kernel setting for elastic
     - name: sysctl
       securityContext:
         privileged: true
         runAsUser: 0
       image: busybox
       command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']

  # Values for data node sets
  data:
    count: 1
    persistence:
      storageClassName: gp2
      size: 100Gi
    resources:
      limits:
        cpu: 1
        memory: 3Gi
    # This privileged initContainer resolves the elasticsearch error:
    # [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
    initContainers:
    # add an init container that adjusts the kernel setting for elastic
     - name: sysctl
       securityContext:
         privileged: true
         runAsUser: 0
       image: busybox
       command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']

Without access to modify the StatefulSets prior to installation, I pointed them to the new ServiceAccount as follows:

kubectl -n logging set serviceaccount sts [ek-es-master|ek-es-data] ek-sa
      serviceAccountName: ek-sa

The combination of applying the StorageClass and RBAC enables the elasticsearch pods to start. The StorageClass allows the elasticsearch-data-ek-es-master-0 & elasticsearch-data-ek-es-data-0 PVCs to bind to PVs and the ek-sa ServiceAccount permits the sysctl container to run.

kubectl -n logging logs ek-es-master-0 -c sysctl
vm.max_map_count = 262144

Note When applying the RBAC configuration after installing EK with flux-bigbang, I was unable to start the pods. Applying the ServiceAccount to the StatefulSets restarts the pods, but I still get unable to validate against any pod security policy errors.