Incompatible with legacy iptables hosts
Summary
Current implementation does not include iptables-legacy which is required for use on hosts that are using iptables instead of nft.
Steps to reproduce
Create a RHEL 7 Kubernetes node using kubeadm. Apply a known working CNI (e.g. open source Calico) Note that Calico and CoreDNS never achieve Ready status
What is the current bug behavior?
Calico and CoreDNS do not achieve a Ready status.
What is the expected correct behavior?
(What you should see instead)
Relevant logs and/or screenshots
kube-proxy logs:
W0126 03:20:04.242266 1 server_others.go:559] Unknown proxy mode "", assuming iptables proxy
I0126 03:20:04.256974 1 node.go:136] Successfully retrieved node IP: xx.xx.xx.xx
I0126 03:20:04.257017 1 server_others.go:186] Using iptables Proxier.
I0126 03:20:04.259406 1 server.go:583] Version: v1.18.14-rc.0
I0126 03:20:04.260621 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0126 03:20:04.260674 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0126 03:20:04.260809 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0126 03:20:04.260889 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0126 03:20:04.262116 1 config.go:315] Starting service config controller
I0126 03:20:04.262146 1 shared_informer.go:223] Waiting for caches to sync for service config
I0126 03:20:04.262191 1 config.go:133] Starting endpoints config controller
I0126 03:20:04.262202 1 shared_informer.go:223] Waiting for caches to sync for endpoints config
I0126 03:20:04.362341 1 shared_informer.go:230] Caches are synced for endpoints config
I0126 03:20:04.362342 1 shared_informer.go:230] Caches are synced for service config
calico-node logs:
2021-01-26 04:23:01.728 [INFO][10] startup/startup.go 454: Hit error connecting to datastore - retry error=Get "https://10.96.0.1:443/api/v1/nodes/foo": dial tcp 10.96.0.1:443: i/o timeout
The other pods, calico-kube-controller and coredns, don't start, but get events:
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c91f6c4edc326e56c4a2fee20746277277099d0d6cb26da2976316aee5649a0a": stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
I went to a node and checked journalctl -u kubelet
. I see the sandbox ones and this:
Jan 26 03:24:30 ip-xx-xx-xx-xx.us-gov-west-1.compute.internal kubelet[2977]: E0126 03:24:30.198382 2977 driver-call.go:266] Failed to unmarshal output for command: init, output: "", error: unexpected end of JSON input
Jan 26 03:24:30 ip-xx-xx-xx-xx.us-gov-west-1.compute.internal kubelet[2977]: W0126 03:24:30.198398 2977 driver-call.go:149] FlexVolume: driver call failed: executable: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~u
Possible fixes
Calico node had similar issues and also relies on iptables for functionality. It was resolved by including /usr/sbin/xtables-legacy-multi
from the open source calico/node image, recreating the symbolic links for iptables[6]-legacy*, and setting up iptables[6] in /etc/alternatives.
dsop/opensource/calico/node#6 (closed)
Defintion of Done
-
Bug has been identified and corrected within the container
/cc @ironbank-notifications/bug