UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects

Custom CAs still not added to trust store

Summary

The Harbor Helm chart (both goharbor community and Big Bang) provide the value caBundleSecretName allowing a user to specify a custom CA bundle. This gets mounted to the container at runtime at /harbor_cust_cert/custom-ca.crt and a script picks it up to add to the default trust store.

Two prior issues have been raised noting this behavior, 88 and 106. 88 was closed because the reporter didn't respond after opening it and the 106 was closed when the install_cert.sh script was fixed to not error out on the fact that the base container layer is now UBI instead of Photon.

However, the install_cert.sh script is not actually run when this container starts. There is an entrypoint.sh script in the container image with the following content:

#!/bin/sh

set -e

/harbor/install_cert.sh

exec /harbor/harbor_core

We can see in the upstream Dockerfile that this is supposed to be the container's entrypoint:

ARG harbor_base_image_version
ARG harbor_base_namespace
FROM ${harbor_base_namespace}/harbor-core-base:${harbor_base_image_version}

HEALTHCHECK CMD curl --fail -s http://localhost:8080/api/v2.0/ping || curl -k --fail -s https://localhost:8443/api/v2.0/ping || exit 1
COPY ./make/photon/common/install_cert.sh /harbor/
COPY ./make/photon/core/entrypoint.sh /harbor/
COPY ./make/photon/core/harbor_core /harbor/
COPY ./src/core/views /harbor/views
COPY ./make/migrations /harbor/migrations
COPY ./icons /harbor/icons

RUN chown -R harbor:harbor /etc/pki/tls/certs \
    && chown -R harbor:harbor /harbor/ \
    && chmod u+x /harbor/entrypoint.sh \
    && chmod u+x /harbor/install_cert.sh \
    && chmod u+x /harbor/harbor_core

WORKDIR /harbor/
USER harbor
ENTRYPOINT ["/harbor/entrypoint.sh"]
COPY make/photon/prepare/versions /harbor/

However, Iron Bank changed this to simply call the harbor_core executable directly, rather than using the entrypoint.sh script:

ARG BASE_REGISTRY=registry1.dso.mil
ARG BASE_IMAGE=ironbank/redhat/ubi/ubi9
ARG BASE_TAG=9.5

FROM goharbor/harbor-core:v2.12.2 as base

FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG}

RUN groupadd -g 1000 harbor && \
    useradd -r -u 1000 -m -s /sbin/nologin -g harbor harbor && \
    dnf upgrade -y --refresh && \
    dnf clean all && \
    rm -rf /var/cache/dnf

COPY --from=base --chown=harbor:harbor /harbor /harbor

RUN sed -i -e 's/Photon/Red Hat/g' /harbor/install_cert.sh
RUN sed -i -e 's/lsb-release/os-release/g' /harbor/install_cert.sh

WORKDIR /harbor
USER harbor

# HEALTHCHECK is no longer part of the OCI format and podman ignores it at build time
#HEALTHCHECK CMD curl --fail -s http://127.0.0.1:8080/api/v2.0/ping || curl -k --fail -s https://127.0.0.1:8443/api/v2.0/ping || exit 1

ENTRYPOINT ["/harbor/harbor_core"]

Thus, while it is possible that the install_cert.sh script will work now, it is never run so custom CA certs still don't get added to the trust store.

Steps to reproduce

Launch this container with custom CAs mounted. If you use the upstream container image, you'll see something like this in the log:

Appending internal tls trust CA to ca-bundle ...
find: '/etc/harbor/ssl': No such file or directory
Internal tls trust CA appending is Done.
Appending trust CA to ca-bundle ...
 /harbor_cust_cert/custom-ca.crt Appended ...
CA appending is Done.
init global config instance failed. If you do not use this, just ignore it.  open conf/app.conf: no such file or directory
2025-03-14T11:06:21Z [INFO] [/controller/artifact/annotation/parser.go:85]: the annotation parser to parser artifact annotation version v1alpha1 registered
2025-03-14T11:06:21Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cncf.helm.config.v1+json registered
2025-03-14T11:06:21Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cnab.manifest.v1 registered

All of the lines without a timestamp are the output of the install_certs.sh script, whereas the entries with a timestamp are logged by the harbor_core executable once it is called.

If you use the Iron Bank container image, you instead see:

init global config instance failed. If you do not use this, just ignore it.  open conf/app.conf: no such file or directory
2025-03-14T17:24:05Z [INFO] [/controller/artifact/annotation/parser.go:85]: the annotation parser to parser artifact annotation version v1alpha1 registered
2025-03-14T17:24:05Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cncf.helm.config.v1+json registered
2025-03-14T17:24:05Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cnab.manifest.v1 registered

You can see it otherwise does the same things, but there is no output from the install_certs.sh script before Harbor starts up because it is never run.

The specific task I am trying to do is migrate a self-hosted Big Bang Harbor that is using the upstream Photon images for a private DoD contractor to use the Iron Bank images, and this installation uses Amazon RDS PostgreSQL as its database and verify-full as the TLS strategy for all of the Harbor components that make connections. Amazon RDS uses CAs that are not included in the Mozilla trust store that is typically what gets bundled with most Linux distributions, so this can only work if the RDS CA is added to the container trust store.

Since this doesn't happen with the Iron Bank image, the Harbor Core and Harbor Job Service components, which are the two that need to connect to the database, are not able to verify the certificate and go into CrashLoopBackOff. The last log you will see when trying to do this is:

2025-03-14T17:34:20Z [INFO] [/common/dao/base.go:67]: Registering database: type-PostgreSQL host-plx-prod-harbor.cacvfbtenubz.us-gov-east-1.rds.amazonaws.com port-5432 database-harborUser sslmode-"verify-full"
[ORM]2025/03/14 17:34:21 Register db Ping `default`, failed to connect to `host=plx-prod-harbor.cacvfbtenubz.us-gov-east-1.rds.amazonaws.com user=harborUser database=harborUser`: failed to write startup message (tls: failed to verify certificate: x509: certificate signed by unknown authority)
2025-03-14T17:34:21Z [FATAL] [/core/main.go:187]: failed to initialize database: Register db Ping `default`, failed to connect to `host=plx-prod-harbor.cacvfbtenubz.us-gov-east-1.rds.amazonaws.com user=harborUser database=harborUser`: failed to write startup message (tls: failed to verify certificate: x509: certificate signed by unknown authority)

What is the current bug behavior?

Harbor starts without ever running its entrypoint script, not adding custom CAs to the container's trust store, causing it to crash if the database has a server cert signed by a custom CA.

What is the expected correct behavior?

Run the entrypoint script and get the custom CAs added.

Relevant logs and/or screenshots

See logs above.

Possible fixes

Change the ENTRYPOINT in the Dockerfile from ["/harbor/harbor_core"] back to ["/harbor/entrypoint.sh"] as it is upstream.

Tasks

  • Bug has been identified and corrected within the container

Please read the Iron Bank Documentation for more info

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

UNCLASSIFIED - NO CUI