IroBank Traefik fails with Local Self Signed Certs where Upstream (Dockerhub) Traefik Works
Summary
TLS termination using self-signed certs in a local environment results in bad certificate errors when using Iron Bank's hardened Traefik instance. The same configuration setup works with the official Traefik image from Docker Hub.
Steps to reproduce
Create a self signed certificates for service endpoints on localhost domain using openssl
openssl req -x509 -sha256 -nodes -keyout {certificate-key} -out {certificate-name} -config {cert configuration} -days 3650 -newkey rsa:2048
Create a volume mount to import the certs into Traefik service defined in docker-compose script (see snippet below):
traefik:
image: "registry1.dso.mil/ironbank/opensource/traefik/traefik:2.5.3"
container_name: "traefik"
command:
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--log.level=DEBUG"
- "--serverstransport.insecureskipverify=true"
- "--api.insecure=true"
- "--api.dashboard=true"
- "--providers.docker=true"
- "--accessLog.filePath=/var/log/access.log"
- "--accessLog.filters.statusCodes=400-499"
- "--accessLog.format=json"
- "--providers.docker=true"
- "--providers.docker.endpoint=unix:///var/run/docker.sock" # Use Docker Socket Proxy instead for improved security
- "--providers.docker.exposedbydefault=false"
- "--providers.file.directory=/rules" # Load dynamic configuration from one or more .toml or .yml files in a directory
- "--providers.file.watch=true" # Only works on top level files in the rules folder
ports:
- "443:443"
- "80:80"
- "8080:8080"
volumes:
- "./traefik_rules:/rules"
**- "./certs:/etc/certs"**
- "/var/run/docker.sock:/var/run/docker.sock:ro"
Create route configuration rules and cert.yml files in the directory specified by the providers.file.directory field:
Dashboard route configuration:
http:
routers:
dashboard:
rule: "Host(`traefik.localhost`)"
service: "api@internal"
entryPoints:
- "websecure"
tls: {}
Application route configuration:
http:
routers:
someapp:
rule: "Host(`someapp.localhost`)"
service: "someapp"
entryPoints:
- "websecure"
tls: {}
services:
someapp:
loadbalancer:
servers:
- url: http://someapp:port/
certs.yml:
tls:
certificates:
- certFile: /etc/certs/someapp.crt
keyFile: /etc/certs/someapp.key
- certFile: /etc/certs/traefik-dashboard.crt
keyFile: /etc/certs/traefik-dashboard.key
(How one can reproduce the issue - this is very important)
What is the current bug behavior?
Visiting the defined routes within a browser result in a 404 page not found error.
What is the expected correct behavior?
Visiting the defined routes within a browser should result in successfully hitting receiving content from the endpoint and displaying the content within the browser.
(What you should see instead)
Relevant logs and/or screenshots
Creating network "bc_default" with the default driver
Creating traefik ... done
Creating bc_step-ca_1 ... done
Creating bc_redis_1 ... done
Creating bc_postgres_1 ... done
Creating bc_migration_1 ... done
Creating bc_bc-app_1 ... done
Attaching to traefik
traefik | time="2022-02-09T16:05:23Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.toml"
traefik | time="2022-02-09T16:05:23Z" level=info msg="Traefik version 2.5.3 built on 2021-09-20T15:43:56Z"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"ping\":{\"address\":\":8082\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"websecure\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\"},\"api\":{\"insecure\":true,\"dashboard\":true},\"ping\":{\"entryPoint\":\"ping\",\"terminatingStatusCode\":503},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"pilot\":{\"dashboard\":true}}"
traefik | time="2022-02-09T16:05:23Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
traefik | time="2022-02-09T16:05:23Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="Start TCP Server" entryPointName=ping
traefik | time="2022-02-09T16:05:23Z" level=debug msg="Start TCP Server" entryPointName=web
traefik | time="2022-02-09T16:05:23Z" level=debug msg="Start TCP Server" entryPointName=traefik
traefik | time="2022-02-09T16:05:23Z" level=debug msg="Start TCP Server" entryPointName=websecure
traefik | time="2022-02-09T16:05:23Z" level=info msg="Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}"
traefik | time="2022-02-09T16:05:23Z" level=info msg="Starting provider *traefik.Provider {}"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="Configuration received from provider internal: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645},\"ping\":{\"entryPoints\":[\"ping\"],\"service\":\"ping@internal\",\"rule\":\"PathPrefix(`/ping`)\",\"priority\":2147483647},\"web-to-websecure\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-websecure\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{},\"ping\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}},\"redirect-web-to-websecure\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"tls\":{}}" providerName=internal
**traefik | time="2022-02-09T16:05:23Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
traefik | time="2022-02-09T16:05:23Z" level=debug msg="http: panic serving 172.19.0.1:58056: runtime error: invalid memory address or nil pointer dereference"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="goroutine 75 [running]:"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="net/http.(*conn).serve.func1()"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="\t/usr/local/golang/1.10.8/go/src/net/http/server.go:1801 +0xb9"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="panic({0x2d787c0, 0x59307b0})"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="\t/usr/local/golang/1.10.8/go/src/runtime/panic.go:1047 +0x266"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="crypto/tls.(*Conn).readClientHello(0xc0000d0700, {0x38ed838, 0xc000488f40})"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="\t/usr/local/golang/1.10.8/go/src/crypto/tls/handshake_server.go:144 +0x7e"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="crypto/tls.(*Conn).serverHandshake(0xc0000d0700, {0x38ed838, 0xc000488f40})"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="\t/usr/local/golang/1.10.8/go/src/crypto/tls/handshake_server.go:43 +0x46"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="crypto/tls.(*Conn).handshakeContext(0xc0000d0700, {0x38ed8e0, 0xc0006ba630})"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="\t/usr/local/golang/1.10.8/go/src/crypto/tls/conn.go:1445 +0x3d1"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="crypto/tls.(*Conn).HandshakeContext(...)"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="\t/usr/local/golang/1.10.8/go/src/crypto/tls/conn.go:1395"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="net/http.(*conn).serve(0xc000493540, {0x38ed8e0, 0xc00084e480})"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="\t/usr/local/golang/1.10.8/go/src/net/http/server.go:1817 +0x230"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="created by net/http.(*Server).Serve"
traefik | time="2022-02-09T16:05:23Z" level=debug msg="\t/usr/local/golang/1.10.8/go/src/net/http/server.go:3033 +0x4e8"**
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Creating middleware" routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Creating middleware" middlewareType=StripPrefix entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal entryPointName=traefik middlewareName=dashboard_redirect@internal
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal middlewareName=tracing
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Creating middleware" middlewareType=Recovery middlewareName=traefik-internal-recovery entryPointName=traefik
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Added outgoing tracing middleware ping@internal" entryPointName=ping routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder
traefik | time="2022-02-09T16:05:24Z" level=debug msg="Creating middleware" entryPointName=ping middlewareName=traefik-internal-recovery middlewareType=Recovery
**traefik | time="2022-02-09T16:05:28Z" level=debug msg="Serving default certificate for request: \"traefik.localhost\""
traefik | time="2022-02-09T16:05:28Z" level=debug msg="http: TLS handshake error from 172.19.0.1:58058: remote error: tls: bad certificate"
traefik | 127.0.0.1 - - [09/Feb/2022:16:05:34 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 1 "ping@internal" "-" 0ms
traefik | 172.19.0.1 - - [09/Feb/2022:16:05:40 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 2 "web-to-websecure@internal" "-" 0ms
traefik | time="2022-02-09T16:05:40Z" level=debug msg="Serving default certificate for request: \"traefik.localhost\""
traefik | time="2022-02-09T16:05:40Z" level=debug msg="http: TLS handshake error from 172.19.0.1:58060: remote error: tls: bad certificate"
traefik | time="2022-02-09T16:05:43Z" level=debug msg="Serving default certificate for request: \"traefik.localhost\""**
traefik | 172.19.0.1 - - [09/Feb/2022:16:05:46 +0000] "GET / HTTP/2.0" - - "-" "-" 3 "-" "-" 0ms
traefik | 172.19.0.1 - - [09/Feb/2022:16:05:46 +0000] "GET /favicon.ico HTTP/2.0" - - "-" "-" 4 "-" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:05:49 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 5 "ping@internal" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:06:05 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 6 "ping@internal" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:06:20 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 7 "ping@internal" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:06:35 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 8 "ping@internal" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:06:50 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 9 "ping@internal" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:07:05 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 10 "ping@internal" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:07:21 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 11 "ping@internal" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:07:36 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 12 "ping@internal" "-" 0ms
traefik | 127.0.0.1 - - [09/Feb/2022:16:07:51 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 13 "ping@internal" "-" 0ms
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)
Possible fixes
N/A (If you can, link to the line of code that might be responsible for the problem)
Defintion of Done
-
Bug has been identified and corrected within the container -
The container functions in the same manner as the official image in Docker Hub or documentation is provided on how the Iron Bank image differs and what needs to be done differently.