UNCLASSIFIED - NO CUI

Skip to content

SPIKE: Enable Secure AWS Credential Access for Helm Chart Testing in k3d Clusters

Problem Statement

We use EC2 instances to spin up k3d clusters to test our Helm-based Kubernetes deployments. This lightweight setup enables fast and repeatable testing. However, several of our Helm charts require access to AWS services (e.g., S3, Secrets Manager, KMS).

With our recent migration to AWS SSO, engineers no longer use long-lived IAM users. Instead, they rely on temporary credentials and browser-authenticated sessions via aws sso login. This introduces a key challenge:

Some Helm charts (e.g., MinIO, External Secrets Operator, GitLab, possibly Nexus) require AWS credentials in the form of static AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. These charts do not support STS tokens or browser-authenticated sessions.

As a result:

  • We cannot run automated testing for these charts inside k3d without provisioning static credentials.
  • We cannot use IRSA, as k3d does not support Kubernetes service accounts with IRSA out of the box (due to the lack of a cloud provider and native OIDC support).

Proposed Exploration: Using k3d-irsa

The k3d-irsa tool is designed to simulate IRSA in local environments by injecting a fake OIDC provider and mimicking the IRSA mechanism inside k3d clusters.

This spike will explore:

  • Whether k3d-irsa is a viable solution for enabling IRSA-compatible AWS authentication without static keys.
  • Whether it can be cleanly integrated into our development process.

Areas to Analyze

1. Setup Complexity

  • What is the effort required to integrate k3d-irsa into our existing EC2/k3d setup?
  • Does it require patching pods or modifying Helm chart values? (It will most likely require adding a Kyverno rule to enable the tool to function properly.)

2. Security Considerations

  • Is there any risk of leaking sensitive tokens or injecting unsafe IAM permissions?
  • Can we scope IAM roles narrowly to avoid overly permissive access?

3. Chart Compatibility

Evaluate specific Helm charts to confirm whether they:

  • Support IRSA (i.e., can be configured to assume roles via service accounts).
  • Can function properly with k3d-irsa’s simulated setup.

Charts to evaluate:

  • External Secrets Operator
  • HashiCorp Vault
  • MinIO (if configured to access an S3 backend)
  • GitLab (object storage, secrets, etc.)
  • Nexus (verify if it integrates with S3 or Secrets Manager)

4. Operational Pain Points

  • How will we manage or rotate the credentials used by k3d-irsa?
  • How will we detect and debug failures within the IRSA simulation layer?

Known Constraints & Risks

  • k3d lacks native support for cloud provider integrations, so this is a workaround that may break under certain Helm chart assumptions.
  • k3d-irsa is an open-source project with potentially limited maintenance or community support.
  • Helm charts may require significant customization to work with a simulated IRSA environment.

Acceptance Criteria for the Spike

This spike will serve as a discussion point to determine whether it is worthwhile to invest time in testing this solution.

If we decide to proceed, the following could become tasks under an Epic to evaluate the implementation:

  • Set up a working k3d-irsa-based cluster on EC2.
  • Deploy and validate at least two Helm charts (e.g., ESO and GitLab) that require IAM roles.
  • Authenticate into AWS services without using static key/secret pairs.
  • Document a repeatable setup process that can be CI-integrated.
  • Identify any gaps or incompatibilities with our Helm charts or tools.

Deliverables

  • Summary of findings
  • Recommendations on adoption path and next steps
Edited by Danilo Patrucco

UNCLASSIFIED - NO CUI