fix: certificate_authority value did not actually trust cert

Relates #70 (closed)

I was able to reproduce the issue by creating a self-signed certificate, assigning it to keycloak, and configuring authservice with global.certificate_authority.

Here is the command to create the self-signed certificate:

openssl req \
  -new \
  -newkey rsa:4096 \
  -days 365 \
  -nodes \
  -x509 \
  -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=keycloak.bigbang.dev" \
  -keyout private-key.pem  \
  -out cert.pem

Here is my yaml override to reproduce the issue:

addons:
  keycloak:
    ingress:
      key: |
        -----BEGIN PRIVATE KEY-----
        MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQDHXeQn4EC/2k7A
        cb3QpRDZ217Z+W4N2BdET5XXF1xhPSyzre6a3xJ3TQiXJY9BmDiLz8t5pc41bki2
        dq5FeU9HtZmmq3/+Z/Tph/bwIKV3aV6x5tqFNMErNcWtnpEN1P0L3kH9ZajWBbji
        cBcO81eKD72qobI03y0uoAQ6rVrK5MQKKGTEM4gZpsx0qkcYSUxJSUF7mKopBbhn
        yAPryP0KtYETiXRepZNrACcafoLcZggzOgJs7vghknieG+oLpd/IR9Uve2yOTIRu
        bSzwjvbzuw+YPpRfGnPlHCkhOU7vRGwS+TjTU15FNlrlkGgouYXX4BtRFzb4dHXH
        DF1YOwdYWrsye5o//FH1tNtUyquRvg4LKwOAnbatClmC6pGQHzDKJD28F1qg0NpA
        UpoJRpEen5em6F3nK4X/Rj8ijJZ+nGBpR/6vcVBcpCIGJawFphDnxLj9XXb05XZ6
        HMlPP8aB4U7T9lLR3XB6RacrQ/PjS403F3oK7TtwjSWEUE0Sc1obADQoubP5yp2G
        3XV6NFJWG2eupgUStJRWDHQu+510EXqyvAmQiWkWSuInxc62TQRxJQB3IYReRMM7
        0MvvNGiLqGHpW9uqL7WWNIRbWRLe42gnVMZ4TO6698MsRMaJG88OJvguqpZfZK+/
        OUaZjIBH7n43qOjd9L1RkR/HCsJquQIDAQABAoICAA4cM7KMTTze4DG/4xeuJRcM
        LwaFoC/H5l6p9NCSOVIzBJPLeL6akOmjaLsc2AqkEcV4bZ3kmad/sJMEa5FkdHNe
        c5tfmjmY5TU1j6k/YEKnMSHtdTQWD6FlpwHingDQ/R9eycQb0rMiGMui9AjVeXG+
        qzV1SAH0tTOOdPorrIg9YHAZG4ek34o0O1Z6+jjqcCUkM4eiPpHUczCaMAfbM+iS
        /8mQkg1EUf6R1lKqV0vux0AL6CHKWIK0xYdoMZyFLwVTay/XABACs6FGphZ5Adqu
        d7w2WqUv/7mckH8m2ZK24KDHpYlUZlDG9/A1NCAEwGeu13QZ7OSxBmhx9DXTSLHM
        yO3c1/aUePWAj/g0/zQIPG5ro9bGUAsIsNQJGuczJ69b8D1WShN1lQMibTnXDZbK
        qPRGvk592TqcfdrKyEYStcNL74DtR2BxH5/fAdbfkr+9RXsS+F70lyYry4P4ud1h
        9GOqQSS1RZJlqkIE5ITfzok17G+U+076Aq3UTbk4oEz5IRlJZhSAd5aOddXhRUf+
        tSjkW6pBUkhXcXdChhehklaWDIMHLmQsqOeTgocaBLbdodwzdx6ocKonGapOV68Z
        zcCsGmqYHgmvcsGomTul/IE/mdRWKywEF48rCuyBLz8CL3JogVeCyjkJeKbVixm3
        kFDE+6JMcfuMEI9V3aKhAoIBAQDr4MVLzJgnAkcWYGlg8egRanjDKNr12s+cu/U5
        KsR2LobBPYg96qtFfxnQnlmeC8S3ueESAr5Wo55H2RY1Mjb2ujNyb+ClpHn+ojgd
        cMyZebxA8YmneaKG6PNnSZ93QzZRTk8ehO1J6PAFLl5eg8LtyXyzenw+LEqEsrvb
        xl3thp3etvmcsthkRUxqjEsyCsz21pAZ7L8j8EXIdhjFLh6pN0h/ECPXp0QlRHl5
        4moZGjGAtkDjcdetwWd66bEitWL1NxtzilAeljw2fAKGr7PIUIshna4iJOq6YzBL
        8uolVbN7e+5lILEAT8wQArgIJNxDu034/Y9uEdxzEt3m9MAZAoIBAQDYX8S9gsdG
        gB8ED+bXQFYACSMC0XSaQdqdQqr0hmWNYMAupyKtHcPSWRbn2OMIhiHoKAJbo7sG
        pCqMeJAsT5kMg9ieQoULOMVRpwTSyYcNwa9TR/Cj3nW6i3g875gJl8y/rfkadXiL
        CsUQ9sPu3eUdHonKJKjiPeQxPYLaOlUMoXkCVExp0WbpQuqFi0bcE+46JSwtulCZ
        ZMGqv94Y1SrqFTl/Ddrm0ItakzAzs7E/2n9sndYpzJ5cQ+xe49rnJ4px89PpIIDN
        f5bXi+XBSnMVnqpJ4VcuoW40Y2mZDNMXoWRcG7jxndUTAJvYExhjNg/lEdUa2gJg
        tbzEd+RAqdOhAoIBAEtSu644512NnKXIo94RYcot9eJcaY+ZEDM80lIvFg2nTeDn
        hVpAVfEbZWL5LiUKHQeOlBDsT1vbY/ANosnZ/zNKwqMIlvROUaa+pqzAGJXqCfOz
        LgINFZl/Sjrxh7rN+8p690kHqb6wY/VDmV8VNGo6rVejBYnVGvbFHLhR4Cy5kDBO
        vecMNY3Pk1dy9ZBFYq5wVhPxRhhz4dTh4YcV8zSWMtBglxGGmHxOIXyKkNbzRluT
        16TJ9dXGrQTTG3257p/fuSRsRql90DyV3TilYwPUtJlHgRPmNg7PHd0i9SJ2+5mz
        Oa2RgvRWhxOVyq7PFoeKiJu6XNMJYRxeKf5WqgECggEAZY11Qljp/H3PkNGBz48j
        jfq70uvJpUi5OUv7/q5BZXx+VqcdEFFDAivMI17ZF7wUl+iHSslq0zAjxDAlwpZN
        R+FtvsGLvfuUKMGR8vihCWTZS+Yc3Fxhtv8UnMz6962mbGRT4QY3YFLasR6QUUEV
        8nfNxfE0zpKr0iyDiLcOCdKTf/Netd+RRHOirLD0vgAgtGuT85ZZk9UFTIWm0NLj
        xF/Xz3kKP4CrlwQ0AGAbFdQQMENYK+Ach0Nd4h9hM1KMmPUOhYE60rhKbLn9SJ4P
        q5r/Y5N+nqUnmzD5c9rgDFw0RULBp60jPzru6hf0P0Q1a5bWBTdDHG7PvQpkN7E9
        YQKCAQAz2xHV25Kma0nMogPnZlScIR65DOxSgZBBuRSIbT768TRivBZWf7vQ7DOw
        WMnvPNa2/Sj8N0SV8DkqrYhPpKflY5Ta/9uK0MxuzocuPyqPqglTSzkjGLNuNsYb
        /EH1PexhUVtXxMyu1PBz/L5WtTDRZ3H7BVRz212zQbXRJ18ki9X+fheLOVeMuSJ+
        9OOCSr8lLWS0+ct3ozZE9rvXTARW5QgjDZhfNW6Z11tH/YkI+0OWaD+FZ6cCTmJE
        JxK9KNOcE0S/g7yRqNxAXsqmC4baND+N2ZxJd9VJtcqCHCgx4Nk4SCbGClDLi8oW
        A8LWaRsK3pcH000jTXGai5+pEnH0
        -----END PRIVATE KEY-----
      cert: |
        -----BEGIN CERTIFICATE-----
        MIIFozCCA4ugAwIBAgIUUnts120I4Mv291dy1tufhsQb5vowDQYJKoZIhvcNAQEL
        BQAwYTELMAkGA1UEBhMCVVMxDzANBgNVBAgMBkRlbmlhbDEUMBIGA1UEBwwLU3By
        aW5nZmllbGQxDDAKBgNVBAoMA0RpczEdMBsGA1UEAwwUa2V5Y2xvYWsuYmlnYmFu
        Zy5kZXYwHhcNMjMwODMwMTg1MTA2WhcNMjQwODI5MTg1MTA2WjBhMQswCQYDVQQG
        EwJVUzEPMA0GA1UECAwGRGVuaWFsMRQwEgYDVQQHDAtTcHJpbmdmaWVsZDEMMAoG
        A1UECgwDRGlzMR0wGwYDVQQDDBRrZXljbG9hay5iaWdiYW5nLmRldjCCAiIwDQYJ
        KoZIhvcNAQEBBQADggIPADCCAgoCggIBAMdd5CfgQL/aTsBxvdClENnbXtn5bg3Y
        F0RPldcXXGE9LLOt7prfEndNCJclj0GYOIvPy3mlzjVuSLZ2rkV5T0e1maarf/5n
        9OmH9vAgpXdpXrHm2oU0wSs1xa2ekQ3U/QveQf1lqNYFuOJwFw7zV4oPvaqhsjTf
        LS6gBDqtWsrkxAooZMQziBmmzHSqRxhJTElJQXuYqikFuGfIA+vI/Qq1gROJdF6l
        k2sAJxp+gtxmCDM6Amzu+CGSeJ4b6gul38hH1S97bI5MhG5tLPCO9vO7D5g+lF8a
        c+UcKSE5Tu9EbBL5ONNTXkU2WuWQaCi5hdfgG1EXNvh0dccMXVg7B1hauzJ7mj/8
        UfW021TKq5G+DgsrA4Cdtq0KWYLqkZAfMMokPbwXWqDQ2kBSmglGkR6fl6boXecr
        hf9GPyKMln6cYGlH/q9xUFykIgYlrAWmEOfEuP1ddvTldnocyU8/xoHhTtP2UtHd
        cHpFpytD8+NLjTcXegrtO3CNJYRQTRJzWhsANCi5s/nKnYbddXo0UlYbZ66mBRK0
        lFYMdC77nXQRerK8CZCJaRZK4ifFzrZNBHElAHchhF5EwzvQy+80aIuoYelb26ov
        tZY0hFtZEt7jaCdUxnhM7rr3wyxExokbzw4m+C6qll9kr785RpmMgEfufjeo6N30
        vVGRH8cKwmq5AgMBAAGjUzBRMB0GA1UdDgQWBBS+FQqvibyRTJx4YzmuoQlhj2vd
        bTAfBgNVHSMEGDAWgBS+FQqvibyRTJx4YzmuoQlhj2vdbTAPBgNVHRMBAf8EBTAD
        AQH/MA0GCSqGSIb3DQEBCwUAA4ICAQAJOUkfapbKCJVS8A6UZwGvu56u8tlhEhgZ
        OZ1WgG6gnNApuerLJvuR2CRBc+whlKKkjnu2ExD3mnVhwnuSNr/xyWaMEq6vN3Tb
        2OUwFSKREDKpFqSuM6iTmPrxCPkHOj/vepMpDEqsqRT4ZZAX5MKJi2MOUEDI8yzh
        Cn/+OHFZAskWd2WGqrWxzOy7zX7WxbQuROd8rft1NZwpZ887IvTqs3ERiW0QRtoh
        uAi8nlBAHOu+tPeVZ4W85bGHkf2KsPusMq3s7bZpP+V4AV1Re66fimEMAacS6X0L
        S4g2k+/U7rMlQXlGK1M4TOEucWrA1xti7aV99IAkF2IFv7UBalLufOnxGpY9gZID
        zCBaCM3+O0HlEIHQeHBey2UHXI8wC9AXmOk2k6tnqqbREReeH/Dh0O7bbJGBY3Ba
        /s7rd3v9k/Toja/7Ty75XCwNkXPustl+sO12GRoNGaq7Scsn3rybmyQRhV1/0kjH
        noj2foowa6cbX/UVN0DdG8Y3hmTdLPz04owrEqW01d68VYI/oP6YP+yUvMfFgiZ9
        l/px/mNglsta6naMWrHmGvbQ4t/qX/NKDAAgCRdPkqlRNY+exkj3Fhk/F+JBeKKu
        906s6knMU77RpCaMvNt6N4pllkV2PYjBtBvLgCH9VpOy03PGIkQKvQ+fyBKj42WY
        uni7Jc2m5Q==
        -----END CERTIFICATE-----

  authservice:
    enabled: true

    values:
      oidc:
        host: keycloak.bigbang.dev

      global:
        certificate_authority: |
          -----BEGIN CERTIFICATE-----
          MIIFozCCA4ugAwIBAgIUUnts120I4Mv291dy1tufhsQb5vowDQYJKoZIhvcNAQEL
          BQAwYTELMAkGA1UEBhMCVVMxDzANBgNVBAgMBkRlbmlhbDEUMBIGA1UEBwwLU3By
          aW5nZmllbGQxDDAKBgNVBAoMA0RpczEdMBsGA1UEAwwUa2V5Y2xvYWsuYmlnYmFu
          Zy5kZXYwHhcNMjMwODMwMTg1MTA2WhcNMjQwODI5MTg1MTA2WjBhMQswCQYDVQQG
          EwJVUzEPMA0GA1UECAwGRGVuaWFsMRQwEgYDVQQHDAtTcHJpbmdmaWVsZDEMMAoG
          A1UECgwDRGlzMR0wGwYDVQQDDBRrZXljbG9hay5iaWdiYW5nLmRldjCCAiIwDQYJ
          KoZIhvcNAQEBBQADggIPADCCAgoCggIBAMdd5CfgQL/aTsBxvdClENnbXtn5bg3Y
          F0RPldcXXGE9LLOt7prfEndNCJclj0GYOIvPy3mlzjVuSLZ2rkV5T0e1maarf/5n
          9OmH9vAgpXdpXrHm2oU0wSs1xa2ekQ3U/QveQf1lqNYFuOJwFw7zV4oPvaqhsjTf
          LS6gBDqtWsrkxAooZMQziBmmzHSqRxhJTElJQXuYqikFuGfIA+vI/Qq1gROJdF6l
          k2sAJxp+gtxmCDM6Amzu+CGSeJ4b6gul38hH1S97bI5MhG5tLPCO9vO7D5g+lF8a
          c+UcKSE5Tu9EbBL5ONNTXkU2WuWQaCi5hdfgG1EXNvh0dccMXVg7B1hauzJ7mj/8
          UfW021TKq5G+DgsrA4Cdtq0KWYLqkZAfMMokPbwXWqDQ2kBSmglGkR6fl6boXecr
          hf9GPyKMln6cYGlH/q9xUFykIgYlrAWmEOfEuP1ddvTldnocyU8/xoHhTtP2UtHd
          cHpFpytD8+NLjTcXegrtO3CNJYRQTRJzWhsANCi5s/nKnYbddXo0UlYbZ66mBRK0
          lFYMdC77nXQRerK8CZCJaRZK4ifFzrZNBHElAHchhF5EwzvQy+80aIuoYelb26ov
          tZY0hFtZEt7jaCdUxnhM7rr3wyxExokbzw4m+C6qll9kr785RpmMgEfufjeo6N30
          vVGRH8cKwmq5AgMBAAGjUzBRMB0GA1UdDgQWBBS+FQqvibyRTJx4YzmuoQlhj2vd
          bTAfBgNVHSMEGDAWgBS+FQqvibyRTJx4YzmuoQlhj2vdbTAPBgNVHRMBAf8EBTAD
          AQH/MA0GCSqGSIb3DQEBCwUAA4ICAQAJOUkfapbKCJVS8A6UZwGvu56u8tlhEhgZ
          OZ1WgG6gnNApuerLJvuR2CRBc+whlKKkjnu2ExD3mnVhwnuSNr/xyWaMEq6vN3Tb
          2OUwFSKREDKpFqSuM6iTmPrxCPkHOj/vepMpDEqsqRT4ZZAX5MKJi2MOUEDI8yzh
          Cn/+OHFZAskWd2WGqrWxzOy7zX7WxbQuROd8rft1NZwpZ887IvTqs3ERiW0QRtoh
          uAi8nlBAHOu+tPeVZ4W85bGHkf2KsPusMq3s7bZpP+V4AV1Re66fimEMAacS6X0L
          S4g2k+/U7rMlQXlGK1M4TOEucWrA1xti7aV99IAkF2IFv7UBalLufOnxGpY9gZID
          zCBaCM3+O0HlEIHQeHBey2UHXI8wC9AXmOk2k6tnqqbREReeH/Dh0O7bbJGBY3Ba
          /s7rd3v9k/Toja/7Ty75XCwNkXPustl+sO12GRoNGaq7Scsn3rybmyQRhV1/0kjH
          noj2foowa6cbX/UVN0DdG8Y3hmTdLPz04owrEqW01d68VYI/oP6YP+yUvMfFgiZ9
          l/px/mNglsta6naMWrHmGvbQ4t/qX/NKDAAgCRdPkqlRNY+exkj3Fhk/F+JBeKKu
          906s6knMU77RpCaMvNt6N4pllkV2PYjBtBvLgCH9VpOy03PGIkQKvQ+fyBKj42WY
          uni7Jc2m5Q==
          -----END CERTIFICATE-----


      chains:
        # some chains here

With this MR, when global.certificate_authority is provided, the following happens:

1. A ca-bundle volume is created.
2. An init container is created that mounts the ca-bundle volume to /certs. The init container copies the system ca-bundle.crt to /certs, and then appends our provided ca to it.
3. The authservice container mounts the ca-bundle volume to /certs, which now contains a ca-bundle.crt with our custom CA appended to it.
4. The authservice container has a new environment variable, SSL_CERT_FILE: /certs/ca-bundle.crt. This tells OpenSSL to use the bundle at /certs/ca-bundle.crt, which allows the authservice container to hit keycloak and properly validate the certificate.

added community-contribution kindbug priority7 labels

changed the description

added 1 commit

• f362c26b - fix: certificate_authority feature did not add cert to root ca bundle

Compare with previous version

marked this merge request as ready

assigned to @ryan.j.garcia, @michaelmartin, @jimmy.bourque, @chris.oconnell, and @kershaw.jacob

@jrb This merge request is not marked as draft, if it is ready for review please add the label, statusreview.

added needs-labels label

added statusreview label

added statusdoing label and removed statusreview label

Taking out of review, we need more testing on this as mentioned in my comment on the issue. We do want to have /etc/pki/tls/certs/ca-bundle.crt be the default most of the time and will want to be careful about when we alter this. Have also done similar testing mentioned above and were not seeing issues.

added 3 commits

• f362c26b...7ed8b599 - 2 commits from branch main
• ec8c4dc7 - fix: certificate_authority feature did not add cert to root ca bundle

Compare with previous version

added 1 commit

• 39049a00 - fix: certificate_authority feature did not add cert to root ca bundle

Compare with previous version

removed needs-labels label

added statusreview label and removed statusdoing label

I added the status::review label back to this issue because I think I have addressed concerns. Is there anything I can do to help this MR move forward?

changed title from Resolve ".Values.global.certificate_authority mount is ignored without additional postRenderer" to fix: certificate_authority value did not actually trust cert

added 1 commit

• 8967251f - fix: certificate_authority value did not actually trust cert

Compare with previous version

requested review from @ryan.j.garcia and @michaelmartin

added 3 commits

• 8967251f...9f13dec3 - 2 commits from branch main
• 3c71dc54 - fix: certificate_authority value did not actually trust cert

Compare with previous version

requested review from @jimmy.bourque

requested review from @chris.oconnell

requested review from @kershaw.jacob

I'm trying to understand this a bit better. Does the provided authservice trusted_certificate_authority value just not work correclty? It seems this would be the authservice expected/provided application-layer level of handling the trust -- instead of depending on openssl's lower level trust using the SSL_CERT_FILE environment variable.

added 1 commit

• 05622652 - run initcontainer as non-root

Compare with previous version

added 1 commit

• 33c83539 - keep original mounts in for minimal changes

Compare with previous version

resolved all threads

added 1 commit

• f8d29559 - remove un-needed mount for initcontainer

Compare with previous version

I just fully tested this again with the new changes. Everything seems to work. This seems good to go to me.

Testing steps:

1. Stood up dev cluster w/ keycloak, default certs, and podinfo protected by authservice. Verified it is all working.
2. Swapped keycloak cert for self-signed cert. Verified authservice was failing to retrieve the JWKs from keycloak -- things are now broken.
3. Passed the self-signed cert into authservice with trusted_certificate_authority. Verified things are all working again.

added 1 commit

• f4335bca - remove unneeded volume mounts

Compare with previous version

added 3 commits

• f4335bca...fdd2bcef - 2 commits from branch main
• 8b9bc58c - Merge remote-tracking branch 'origin/main' into

Compare with previous version

changed the description

@michaelmartin and I seem to have resolved all of the mentioned concerns (we are no longer running any containers as root), and we've simplified the implementation of this fix quite a bit. I think this is ready to merge. Can we get another review?

As the author of !110 (merged), I approve. I understand why this was needed, and am in fact a bit confused how my change worked at all. The magic of CA trust. I unfortunately cannot fully replicate this at the moment, but I will report back once my platform is on whatever version of Big Bang this ends up in (2.11?).

changed milestone to %2.11.0

approved this merge request

changed the description

mentioned in commit 4710ac4f

merged

mentioned in merge request big-bang/bigbang!3124 (merged)