Trouble upgrading gitlab to 14.7.1 "fe_sendauth: error sending password authentication"

My gut says FIPS, but I can't track down for sure how. I'm working through upgrade issues on gitlab from P1 helm chart. The version installed is 14.6.2. The version flux is trying to get us to is 14.7.1, just released yesterday. (Thank you!)

Symptoms:

• I can psql into the DB (RDS instance) from my own machine. (psql 12.9)
• gitlab-sidekiq container 14.6.2 works in cluster with psql 13.2.
• gitlab-sidekiq container 14.7.1 fails in cluster with pqsl 14.1.

The gitlab sidekiq initContainer (dependencies) on 14.7.1 is tossing "fe_sendauth: error sending password authentication" when verifying database connectivity causing the pod to fail init. The webserver pods are doing the same thing on the same init step.

I tracked down that exact error message to postgres' fe_auth.c code (line 1046). Looking at the pg_password_sendauth() function (line 771) that seems to be returning a not ok response. If I'm reading things correctly, pg_password_sendauth() should be receiving a non-MD5 password as an AUTH_REQ_PASSWORD authorization request and setting pwd_to_send. I'm not understanding how we're getting into the state which triggers that error message.

The scripts and versions of ruby, openssl in the two containers are identical. Even all the gems are the same. The only difference I can find between the containers is the version of psql installed.

I also checked and made sure the node the pods are running on doesn't make a difference so it isn't a networking issue or something like that.

Any idea why I'm getting "fe_sendauth: error sending password authentication"?