Cannot use hardened rook/cassandra:v1.5.11

Summary

My goal is to deploy Rook Cassandra nodes using images in Ironbank in AWS.

BONUS: Enable istio injection on each namespace from the latest BigBang release.

registry1.dso.mil/ironbank/rook/cassandra:v1.5.11

Approach

1. Deploy rook-cass-operator using operator.yaml and the rook/cassandra:v1.5.11 image from Ironbank
2. Deploy rook-cass-cluster using cluster.yaml and the cassandra:3.11.6 image

Note: rook-cass-cluster also uses rook/cassandra:v1.5.11 in an initContainer called rook-install

Steps to reproduce

1. Create a Namespace and Secret with Ironbank registry creds 2 Add imagePullSecrets to operator.yaml
2. Apply https://repo1.dso.mil/dsop/rook/cassandra/-/blob/development/deployment/operator.yaml
3. Create another Namespace and Secret with Ironbank registry creds
4. Create storage class and specify name in cluster.yaml
5. Apply https://repo1.dso.mil/dsop/rook/cassandra/-/blob/development/deployment/cluster.yaml

What is the current bug behavior?

Note: rook-cass-operator and rook-cass-cluster deploy successfully with unhardened Internet images.

TEST CASE 1:

• rook-cass-operator: registry1.dso.mil/ironbank/rook/cassandra:v1.5.11 (Istio injection disabled)
• rook-cass-cluster (rook-install container): registry1.dso.mil/ironbank/rook/cassandra:v1.5.11 (Istio injection enabled)
• rook-cass-cluster (cassandra container): cassandra:3.11.6 (unavailable in Ironbank)

Note: By enabling istio injection on the rook-cass-cluster namespace, the private-registry imagePullSecret is appended to the manifest so I can pull from Ironbank in the rook-install initContainer (otherwise there is no imagePullSecret).

Using the Ironbank imagePullSecret, the rook-cass-operator successfully deploys using the hardened image.

Problem: User 1001 does not have permission to execute a 'cp' command in rook-install initContainer.

cp: cannot open '/sidecar/plugins/jmx_prometheus.jar' for reading: Permission denied
cp: cannot open '/sidecar/plugins/jolokia.jar' for reading: Permission denied
cp: cannot open '/sidecar/tini' for reading: Permission denied

Copy command found in rook/pkg/operator/cassandra/controller/util/resource.go

TEST CASE 2:

• rook-cass-operator: registry1.dso.mil/ironbank/rook/cassandra:v1.5.11 (Istio injection enabled)
• rook-cass-cluster (rook-install container): registry1.dso.mil/ironbank/rook/cassandra:v1.5.11 (Istio injection enabled)
• rook-cass-cluster (cassandra container): cassandra:3.11.6 (unavailable in Ironbank)

Problem: Istio injection on rook-cass-operator namespace causes error below.

2021-05-31 15:01:28.587945 I | rookcmd: starting Rook v1.5.11 with arguments '/usr/local/bin/rook cassandra operator'
2021-05-31 15:01:28.588074 I | rookcmd: flag values: --help=false, --log-flush-frequency=5s, --log-level=INFO, --operator-image=, --service-account=
2021-05-31 15:01:28.588078 I | cassandracmd: starting cassandra operator
failed to get container image: failed to find image for container

Problem: How can cassandra:3.11.6 be pulled from anywhere except the Internet and is there an effort for it be hardened and placed in Ironbank?

I have been unsuccessful adjusting the following parameters to create an imagePullSecret and change where the image is being pulled. How can cassandra:3.11.6 be pulled from a custom registry?

cluster.yaml

  mode: cassandra
  version: "3.11.6"

What is the expected correct behavior?

Deploy single replica of rook-cass-operator and any number of rook-cass-cluster nodes using Ironbank images and Istio injection on both namespaces.

Relevant logs and/or screenshots

Relevant portions of logs are displayed above, but full logs can be reproduced at your request.

Possible fixes

Suggestion for test case 1: Make /sidecar recursively readable by user 1001. Afterward, testing may produce new errors to be resolved.

Defintion of Done

• Bug has been identified and corrected within the container