cilium compatability
Feature Request
Why
BB supports CNIs which are compatible with k8s network policies, but Calico has an issue with how it handles ipBlock
. Below is a more detailed writeup of the issue:
Basically, anything that needs access to the internal K8s API is blocked due to how BB handles this netpol and how Cilium handles ipBlock
. Cilium has a known issue with using ipBlock in reference to pod IPs. This appears to actively block pod-to-pod traffic if it is defined at all, so even the 0.0.0.0/0 default value for BB egress API rule renders traffic blocked.
In proving this out, we can see that cilium is dropping traffic from the Istio operator to the default Kubernetes service. The request from the operator is to the actual service address (typically 10.96.0.1:443) but Cilium blocks it as it is forwarded on to the svc endpoint of the nodes on port 6443 in our case.
jtaylor@ubuntu-server:~$ k get po -n istio-operator -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
istio-operator-6b4b7d46d-kcdxn 1/1 Running 2 (39s ago) 101s 10.42.3.218 baorke2-k8s-w-2 <none> <none>
jtaylor@ubuntu-server:~$
jtaylor@ubuntu-server:~$
jtaylor@ubuntu-server:~$ k exec -n kube-system cilium-fz9r7 -- cilium monitor --type drop
Defaulted container "cilium-agent" out of: cilium-agent, install-portmap-cni-plugin (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
xx drop (Policy denied) flow 0xcb1eafd9 to endpoint 0, file bpf_lxc.c line 1181, , identity 41561->kube-apiserver: 10.42.3.218:59490 -> 10.10.16.4:6443 tcp SYN
xx drop (Policy denied) flow 0xf19567eb to endpoint 0, file bpf_lxc.c line 1181, , identity 41561->kube-apiserver: 10.42.3.218:59490 -> 10.10.16.4:6443 tcp SYN
jtaylor@ubuntu-server:~$
jtaylor@ubuntu-server:~$
jtaylor@ubuntu-server:~$ k get netpol -n istio-operator allow-egress-api-istio-operator -o yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
annotations:
meta.helm.sh/release-name: istio-operator-istio-operator
meta.helm.sh/release-namespace: istio-operator
creationTimestamp: "2023-03-23T14:07:34Z"
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
helm.toolkit.fluxcd.io/name: istio-operator
helm.toolkit.fluxcd.io/namespace: bigbang
name: allow-egress-api-istio-operator
namespace: istio-operator
resourceVersion: "36890"
uid: 9466f323-2372-45f3-b721-5ede41017917
spec:
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32
podSelector: {}
policyTypes:
- Egress
status: {}
In trying to use existing BB values, we did attempt to also specify the host cidr but any value in ipBlock appears to deny pod-to-pod traffic. In our case, we solved this in a BB-native manner with a postRenderer in the main values file:
postRenderers:
- kustomize:
patchesJson6902:
- target:
group: networking.k8s.io
version: v1
kind: NetworkPolicy
name: allow-egress-api
patch:
- op: remove
path: /spec/egress/0/to
- op: add
path: /spec/egress/0/ports
value:
- port: 6443
protocol: TCP
https://docs.cilium.io/en/latest/network/kubernetes/policy/#networkpolicy-state Known missing features for Kubernetes Network Policy: ipBlock set with a pod IP cilium/cilium#9209 ipBlock does not work with pod (or node?) IPs in Cilium. This makes it impossible to write a NetworkPolicy to allowlist traffic to the k8s api server (which is a (headless?) Service that points to the node IPs?). BB currently provides two configurations that either:
- deny only 169.254.169.254/32 (allow 0.0.0.0/0) o This is the default BB configuration
- or allowlist a master node CIDR range Both of these configurations specify an ipBlock which is incompatible with Cilium and actually results in blocking all cluster internal network traffic. A Cilium specific CiliumNetworkPolicy alternative may work cilium/cilium#20550. This is difficult to fit into BB because it is specific to one CNI. Action: Update BB documentation? • The deny 169.254.169.254 NetworkPolicy is incompatible with Cilium because it uses ipBlock • This policy may be removed if you have an alternative method to block 169.254.169.254 • Or, the below CiliumNetworkPolicy may be used instead to allowlist communication to the k8s api server
kind: CiliumNetworkPolicy
metadata:
name: cilium-api-server-egress
namespace: default
spec:
endpointSelector:
matchLabels: {} # all pods
egress:
- toEntities:
- kube-apiserver
- toPorts:
- ports:
- port: "6443"
protocol: TCP
Proposed Solution
This solutions could be handled via postRenderer workarounds as shown above or with perhaps another (unsupported?) addon package for Cilium users. This package could be a helm chart with a Cilium Net Policy (CNP) per effected namespace/pod. This appears to always take precedence over the broken ipBlock
k8s netpol. Below is a snippet of what that manifest would look like:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: cilium-api-server-egress
namespace: kyverno
spec:
endpointSelector:
matchLabels:
whateverlabel: formysourcepod
egress:
- toEntities:
- kube-apiserver
- toPorts:
- ports:
- port: "6443" # perhaps a configurable variable as some distros/managed k8s do not use 6443 or removed entirely with toEntities sufficing
protocol: TCP