Calico node pod fails to program iptables using iptables-restore due to an invalid argument

Summary

Calico node pod fails to program iptables using iptables-restore due to an invalid argument. The configuration is from projectcalico. The same steps using the open source calico/node:v3.17.0 image has no issues.

Steps to reproduce

1. Download https://docs.projectcalico.org/v3.17/manifests/calico.yaml
2. Modify to use approved Iron Bank Calico v3.17.0 images
3. Check logs for calico-node-xxxxx

What is the current bug behavior?

Pods fail readiness checks. Logs (below) indicate invalid arguments to iptables-restore.

What is the expected correct behavior?

Pod successfully modifies iptables.

Relevant logs and/or screenshots

2021-01-09 20:23:15.212 [INFO][8748] felix/table.go 596: Loading current iptables state and checking it is correct. ipVersion=0x4 table="raw"
2021-01-09 20:23:15.245 [WARNING][8748] felix/table.go 1314: Failed to execute ip(6)tables-restore command error=exit status 4 errorOutput="iptables-restore v1.8.4 (nf_tables): \nline 14: RULE_INSERT failed (Invalid argument): rule in chain PREROUTING\nline 15: RULE_INSERT failed (Invalid argument): rule in chain OUTPUT\n" input="*raw\n:cali-from-host-endpoint - -\n:cali-to-host-endpoint - -\n:cali-PREROUTING - -\n:cali-OUTPUT - -\n-A cali-OUTPUT -m comment --comment \"cali:njdnLwYeGqBJyMxW\" --jump MARK --set-mark 0/0xf0000\n-A cali-OUTPUT -m comment --comment \"cali:rz86uTUcEZAfFsh7\" --jump cali-to-host-endpoint\n-A cali-OUTPUT -m comment --comment \"cali:pN0F5zD0b8yf9W1Z\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-PREROUTING -m comment --comment \"cali:XFX5xbM8B9qR10JG\" --jump MARK --set-mark 0/0xf0000\n-A cali-PREROUTING -m comment --comment \"cali:EWMPb0zVROM-woQp\" --in-interface cali+ --jump MARK --set-mark 0x40000/0x40000\n-A cali-PREROUTING -m comment --comment \"cali:V6ooGP15glg7wm91\" -m mark --mark 0x40000/0x40000 -m rpfilter --invert --jump DROP\n-A cali-PREROUTING -m comment --comment \"cali:RMTzKqp0j735XfY4\" -m mark --mark 0/0x40000 --jump cali-from-host-endpoint\n-A cali-PREROUTING -m comment --comment \"cali:T8-Zfumo2dKygI73\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-I PREROUTING -m comment --comment \"cali:6gwbT8clXdHdC1b1\" --jump cali-PREROUTING\n-I OUTPUT -m comment --comment \"cali:tVnHkvAo15HuiPy0\" --jump cali-OUTPUT\nCOMMIT\n" ipVersion=0x4 output="" table="raw"
2021-01-09 20:23:15.245 [ERROR][8748] felix/table.go 996: Failed to program iptables, loading diags before panic. error=exit status 4 ipVersion=0x4 table="raw"
2021-01-09 20:23:15.247 [ERROR][8748] felix/table.go 1002: Current state of iptables ipVersion=0x4 iptablesState="# Generated by iptables-save v1.8.4 on Sat Jan  9 20:23:15 2021\n*raw\nCOMMIT\n# Completed on Sat Jan  9 20:23:15 2021\n" table="raw"
2021-01-09 20:23:15.248 [PANIC][8748] felix/table.go 1004: Failed to program iptables, giving up after retries error=exit status 4 ipVersion=0x4 table="raw"
panic: (*logrus.Entry) 0xc000a54640

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Defintion of Done

• Bug has been identified and corrected within the container

/cc @ironbank-notifications/bug