UNCLASSIFIED - NO CUI

Skip to content

cilium compatability

Feature Request

Why

BB supports CNIs which are compatible with k8s network policies, but Calico has an issue with how it handles ipBlock. Below is a more detailed writeup of the issue:

Basically, anything that needs access to the internal K8s API is blocked due to how BB handles this netpol and how Cilium handles ipBlock. Cilium has a known issue with using ipBlock in reference to pod IPs. This appears to actively block pod-to-pod traffic if it is defined at all, so even the 0.0.0.0/0 default value for BB egress API rule renders traffic blocked.

In proving this out, we can see that cilium is dropping traffic from the Istio operator to the default Kubernetes service. The request from the operator is to the actual service address (typically 10.96.0.1:443) but Cilium blocks it as it is forwarded on to the svc endpoint of the nodes on port 6443 in our case.

jtaylor@ubuntu-server:~$ k get po -n istio-operator -o wide
NAME                             READY   STATUS    RESTARTS      AGE    IP            NODE              NOMINATED NODE   READINESS GATES
istio-operator-6b4b7d46d-kcdxn   1/1     Running   2 (39s ago)   101s   10.42.3.218   baorke2-k8s-w-2   <none>           <none>
jtaylor@ubuntu-server:~$
jtaylor@ubuntu-server:~$
jtaylor@ubuntu-server:~$ k exec -n kube-system cilium-fz9r7 -- cilium monitor --type drop
Defaulted container "cilium-agent" out of: cilium-agent, install-portmap-cni-plugin (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
xx drop (Policy denied) flow 0xcb1eafd9 to endpoint 0, file bpf_lxc.c line 1181, , identity 41561->kube-apiserver: 10.42.3.218:59490 -> 10.10.16.4:6443 tcp SYN
xx drop (Policy denied) flow 0xf19567eb to endpoint 0, file bpf_lxc.c line 1181, , identity 41561->kube-apiserver: 10.42.3.218:59490 -> 10.10.16.4:6443 tcp SYN
jtaylor@ubuntu-server:~$
jtaylor@ubuntu-server:~$
jtaylor@ubuntu-server:~$ k get netpol -n istio-operator allow-egress-api-istio-operator -o yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  annotations:
    meta.helm.sh/release-name: istio-operator-istio-operator
    meta.helm.sh/release-namespace: istio-operator
  creationTimestamp: "2023-03-23T14:07:34Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: istio-operator
    helm.toolkit.fluxcd.io/namespace: bigbang
  name: allow-egress-api-istio-operator
  namespace: istio-operator
  resourceVersion: "36890"
  uid: 9466f323-2372-45f3-b721-5ede41017917
spec:
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 169.254.169.254/32
  podSelector: {}
  policyTypes:
  - Egress
status: {}

In trying to use existing BB values, we did attempt to also specify the host cidr but any value in ipBlock appears to deny pod-to-pod traffic. In our case, we solved this in a BB-native manner with a postRenderer in the main values file:

  postRenderers:
  - kustomize:
      patchesJson6902:
      - target:
          group: networking.k8s.io
          version: v1
          kind: NetworkPolicy
          name: allow-egress-api
        patch:
          - op: remove
            path: /spec/egress/0/to
          - op: add
            path: /spec/egress/0/ports
            value:
              - port: 6443
                protocol: TCP

https://docs.cilium.io/en/latest/network/kubernetes/policy/#networkpolicy-state Known missing features for Kubernetes Network Policy: ipBlock set with a pod IP cilium/cilium#9209 ipBlock does not work with pod (or node?) IPs in Cilium. This makes it impossible to write a NetworkPolicy to allowlist traffic to the k8s api server (which is a (headless?) Service that points to the node IPs?). BB currently provides two configurations that either:

  1. deny only 169.254.169.254/32 (allow 0.0.0.0/0) o This is the default BB configuration
  2. or allowlist a master node CIDR range Both of these configurations specify an ipBlock which is incompatible with Cilium and actually results in blocking all cluster internal network traffic. A Cilium specific CiliumNetworkPolicy alternative may work cilium/cilium#20550. This is difficult to fit into BB because it is specific to one CNI. Action: Update BB documentation? • The deny 169.254.169.254 NetworkPolicy is incompatible with Cilium because it uses ipBlock • This policy may be removed if you have an alternative method to block 169.254.169.254 • Or, the below CiliumNetworkPolicy may be used instead to allowlist communication to the k8s api server
kind: CiliumNetworkPolicy
metadata:
  name: cilium-api-server-egress
  namespace: default
spec:
  endpointSelector:
    matchLabels: {} # all pods
  egress:
  - toEntities:
    - kube-apiserver
  - toPorts:
    - ports:
      - port: "6443"
        protocol: TCP

Proposed Solution

This solutions could be handled via postRenderer workarounds as shown above or with perhaps another (unsupported?) addon package for Cilium users. This package could be a helm chart with a Cilium Net Policy (CNP) per effected namespace/pod. This appears to always take precedence over the broken ipBlock k8s netpol. Below is a snippet of what that manifest would look like:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cilium-api-server-egress
  namespace: kyverno
spec:
  endpointSelector:
    matchLabels:
      whateverlabel: formysourcepod
  egress:
  - toEntities:
    - kube-apiserver
  - toPorts:
    - ports:
      - port: "6443" # perhaps a configurable variable as some distros/managed k8s do not use 6443 or removed entirely with toEntities sufficing
        protocol: TCP

UNCLASSIFIED - NO CUI