EK DNS netpol does not allow port 5353 for Openshift

assigned to @ryan.j.garcia and @kenna81

@dyoung do you have a value set for networkPolicies.controlPlaneCidr ? As well as openshift.enabled=true within the bigbang chart values?

@dyoung we'll need some info if you will be able to exec into the pods and run some curl commands, along with what openshift version you're running. If you can exec into one of the logging-ek-es-data-# pods and try and curl one of those endpoints it says "Name or service not known" for and verify in-cluster DNS resolution is truly blocked.

That issue above has been seen before but was fixed in versions of the elasticsearch-kibana package a few weeks ago.

We have a specific network policy to allow DNS communication specifically to alleviate this issue, please ensure this networkpolicy is present when networkPolicies are enabled: https://repo1.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana/-/blob/main/chart/templates/bigbang/networkpolicies/kube-dns-egress.yaml

@ryan.j.garcia we don't have openshift.enabled=true, but we have openshift: true, which is how we've always had it.

looks like we're using the default of controlPlaneCidr: 0.0.0.0/0

We are using openshift version 4.6.4.

I can definitely exec in and run some curl tests. I'll look for that netpol now and try some curls to verify DNS access

Looks like DNS is blocked:

[istio-proxy@logging-ek-es-data-0 /]$ curl -G logging-ek-es-data-0.logging-ek=es=data.logging.svc
curl: (6) Could not resolve host: logging-ek-es-data-0.logging-ek=es=data.logging.svc

And I do see that allow-dns-egress netpol in my cluster's logging ns:

spec:
  podSelector: {}
  egress:
    - ports:
        - protocol: UDP
          port: 53
      to:
        - namespaceSelector: {}
  policyTypes:
    - Egress

We've identified that the issue is Openshift using 5353 for DNS instead of 53, updating the two DNS rules allow all the eks pods to come up succesfully

changed title from eks pods having trouble with network policies on BB 1.11.0 with openshift to EK DNS netpol does not allow port 5353 for Openshift