FIPS issue with efs-csi driver
This was a ticket sent to the developers of the efs-csi driver that were referred to in the documentation to create the storage class. https://repo1.dso.mil/big-bang/bigbang/-/blob/master/docs/prerequisites/default-storageclass.md
To sum up the material below, we are running into an issue where their verison with the FIPS flag is not using the fips endpoint. Because of this we're getting a certificate error that wouldn't occur if it was using the fips endpoint. Is there another driver you reccommend or a fix for this issue to be able to deploy Big Bang in AWS, using an EFS StorageClass?
The storage class we are using is below, which you may notice is a modified version of the one provided by the Big Bang documentation.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2
annotations:
storageclass.kubernetes.io/is-default-class: 'true'
provisioner: efs.csi.aws.com
mountOptions:
- tls
parameters:
type: gp2 #gp3 isn't supported by the in-tree plugin
provisioningMode: efs-ap
fileSystemId: <REDACTED>
# fsType: ext4"
# encrypted: 'true' #requires kubernetes nodes have IAM rights to a KMS key
# kmsKeyId: 'arn:aws-us-gov:kms:us-gov-west-1:110518024095:key/b6bf63f0-dc65-49b4-acb9-528308195fd6'
reclaimPolicy: Retain
allowVolumeExpansion: true
helm upgrade --set useFips=true --install aws-efs-csi-driver --namespace kube-system aws-efs-csi-driver/aws-efs-csi-driver is still hitting the default https://elasticfilesystem.<REGION>.amazonaws.com/2015-02-01/file-systems?FileSystemId=<REDACTED> endpoint instead of the https://elasticfilesystem-fips.<REGION>.amazonaws.com/2015-02-01/file-systems?FileSystemId=<REDACTED> endpoint.
**What you expected to happen?**
That the helm upgrade command with the --set useFips=true flag would hit the https://elasticfilesystem-fips.<REGION>.amazonaws.com/2015-02-01/file-systems?FileSystemId=<REDACTED> endpoint
**How to reproduce it (as minimally and precisely as possible)?**
Proceed with the installation instructions through helm and then start up the system with helm upgrade --set useFips=true --install aws-efs-csi-driver --namespace kube-system aws-efs-csi-driver/aws-efs-csi-drive
**Anything else we need to know?**:
You will see that the error shown is that there was a certificate error, but this is because its using the wrong endpoint, that error won't occur if it uses the elasticfilesystem-fips endpoint. The issue here is that starting up the system with the fips flag isn't making it use the fips endpoint.
$ helm upgrade --set useFips=true --install aws-efs-csi-driver --namespace kube-system aws-efs-csi-driver/aws-efs-csi-driver
Release "aws-efs-csi-driver" does not exist. Installing it now.
NAME: aws-efs-csi-driver
LAST DEPLOYED: Fri Mar 17 21:24:14 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that aws-efs-csi-driver has started, run:
kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-efs-csi-driver,app.kubernetes.io/instance=aws-efs-csi-driver"
**Environment**
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.16+rke2r1", GitCommit:"60e5135f758b6e43d0523b3277e8d34b4ab3801f", GitTreeState:"clean", BuildDate:"2023-01-18T22:12:37Z", GoVersion:"go1.19.5 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.16+rke2r1", GitCommit:"60e5135f758b6e43d0523b3277e8d34b4ab3801f", GitTreeState:"clean", BuildDate:"2023-01-18T22:12:37Z", GoVersion:"go1.19.5 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
$ helm version
version.BuildInfo{Version:"v3.11.1", GitCommit:"293b50c65d4d56187cd4e2f390f0ada46b4c4737", GitTreeState:"clean", GoVersion:"go1.18.10"}
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
aws-efs-csi-driver kube-system 1 2023-03-17 21:24:14.364574623 +0000 UTC deployed aws-efs-csi-driver-2.4.1 1.5.4
Running pods associated with efs-ci:
efs-csi-controller-f69d4b896-4fv8w 3/3 Running 0 20m
efs-csi-controller-f69d4b896-vn7zl 3/3 Running 0 20m
efs-csi-node-59rrt 3/3 Running 0 20m
efs-csi-node-62kvh 3/3 Running 0 20m
efs-csi-node-n9cwt 3/3 Running 0 20m
efs-csi-node-wf2x7 3/3 Running 0 20m
efs-csi-node-z62fr 3/3 Running 0 20m
**Please also attach debug logs to help us better diagnose**
$ kubectl logs pod/efs-csi-controller-f69d4b896-vn7zl -n kube-system -c efs-plugin
File System failed: RequestError: send request failed
caused by: Get "https://elasticfilesystem.<REGION>.amazonaws.com/2015-02-01/file-systems?FileSystemId=<REDACTED>": x509: certificate signed by unknown authority
E0317 21:43:37.141363 1 driver.go:103] GRPC error: rpc error: code = Internal desc = Failed to fetch File System info: Describe File System failed: RequestError: send request failed
caused by: Get "https://elasticfilesystem.<REGION>.amazonaws.com/2015-02-01/file-systems?FileSystemId=<REDACTED>": x509: certificate signed by unknown authority