## Description ### Enterprise vs. OSS Chart Decision Starting at chart version 7.0.0, Grafana split the enterprise (Grafana Enterprise Logs) and open-source Loki charts into two independent repositories. This requires Big Bang to formally choose which chart to target going forward. The current 6.x chart includes enterprise Loki (GEL) capabilities, though Big Bang has not taken a formal stance on GEL support. The `enterprise.enabled` value defaults to `false` and GEL-specific configuration has not been actively maintained or tested as part of the BB package. The chart split at v7.0.0 is what forces this implicit posture to become an explicit decision. The decision to target the OSS community chart exclusively is driven by: - No active GEL feature usage in current BB deployments - Reduced maintenance surface (one chart instead of two) - Alignment with DoD OSS preference - Operators requiring GEL can deploy via the BB package template process This decision is recorded here as a permanent reference to avoid revisiting the question in future maintenance cycles. No operator communication is required beyond a standard CHANGELOG entry noting that enterprise chart support has been removed. ### Modernization Plan The Loki Big Bang package has been running on the `grafana/loki` enterprise Helm chart, which bundles Grafana Enterprise Logs resources the BB community does not use. Grafana transferred canonical OSS Loki chart ownership to the `grafana-community/helm-charts` repository starting at chart version 7.0.0 (13.x series). The enterprise chart is no longer the appropriate upstream source for a BB OSS deployment. This epic modernizes the Loki package across three sequential phases. The first phase establishes a formal decision record on whether operators should migrate in-place or redeploy when moving from the enterprise chart to the community chart, then completes the migration itself and ensures the BB umbrella chart stays aligned. The second phase converts the package to the BB passthrough/wrapper pattern, eliminating the per-release manual merge overhead of carrying upstream templates. The third phase expands object storage flexibility by adding Garage as a lightweight alternative to the MinIO operator stack. Each phase builds on the previous. The decision record and migration must be stable before the rearchitecture begins, and the rearchitecture should be complete before Garage is introduced as a new subchart dependency. ## Customer or Stakeholder Sponsor There is not a specific Big Bang customer sponsoring this work. The primary stakeholders are the BB Observability team, who own Loki package maintenance and carry the per-release overhead of the current fork pattern, and BB enterprise operators who need a clear and safe path to move off the enterprise chart. Secondary stakeholders are programs running at the edge or on resource-constrained hardware who have expressed interest in a lighter object storage option than the MinIO operator stack. ## Value to User of BB **Operator safety:** The migration from the enterprise chart to the community chart spans seven upstream major versions. Without a formal decision record and documented operator path, teams risk silent data loss or Helm failures requiring manual intervention. This epic gives operators a tested, documented procedure before they attempt the transition. **Platform stability:** Several configuration values and deployment mode names diverged between the enterprise and community chart. Left unreconciled in the BB umbrella chart, these divergences will cause runtime failures for operators who upgrade their cluster without also updating BB. Catching and fixing them as part of this migration prevents unexpected breakage downstream. **BB integration confidence:** The package must be verified against the full BB hardening stack after migrating to a new upstream source. Operators need assurance that istio, kyverno, bb-common network policies, and gluon test scaffolding all function correctly against the community chart before the migration ships. **Reduced maintenance overhead:** The Loki package currently carries approximately 138 upstream template files that require manual review and merge on every kpt update. The passthrough rearchitecture eliminates this entirely for releases that require no BB-specific template changes. The community chart is already architected with thin component templates delegating to shared helper partials, making this the lowest-friction moment to complete the conversion. **Deployment flexibility:** Garage is a single-binary, S3-compatible object store that runs on 1 GB RAM with no operator or controller requirement. Adding it as an optional subchart gives edge and resource-constrained programs a supported, IronBank-backed alternative to the MinIO operator stack. **OKR alignment:** This epic advances package currency (moving off a deprecated upstream source), security posture (reconciling diverged configuration before it causes failures), internal efficiency (reducing per-release template maintenance), and platform breadth (expanding supported storage backends for constrained environments). **Metrics:** - Template files requiring manual review per Loki release (target: zero after passthrough) - Operator-reported upgrade failures on the enterprise-to-community transition (target: zero, via documented procedure) - Time from Garage IronBank image availability to BB package ship (target: one sprint) ## Requirements/Scope ### Migration vs. Redeploy Decision Record Before any implementation work begins, evaluate and formally document whether operators upgrading from the enterprise chart to the community chart should perform an in-place Helm upgrade or a full redeploy of the Loki release. The evaluation must assess the scope of breaking changes across seven upstream major versions, the compatibility of any in-place path with Kubernetes constraints on running workloads, and the data safety profile of each approach given that Loki stores log data in object storage. The output is a decision record committed to `docs/DEVELOPMENT_MAINTENANCE.md` as a permanent operator reference, including the chosen path, the reasoning behind it, and step-by-step instructions for executing the transition safely. ### Community Chart Migration Migrate the package source from `grafana/loki` (6.55.0) to `grafana-community/helm-charts` (13.x series), implementing the operator path established in the decision record. Scope includes removal of all enterprise and GEL templates, values, and image references; reconciliation of any value paths and behavioral defaults that changed between the two charts; MinIO subchart alias migration with backward-compatible value support; and full verification that the package integrates correctly with the BB hardening stack including bb-common network policies, istio service mesh resources, kyverno policy compatibility, and gluon test scaffolding. All helm unit tests and CI pipelines must pass for both monolith and scalable deployment modes. ### BB Umbrella Chart Alignment Update the BB umbrella chart to reflect the breaking configuration changes introduced by the community chart migration. Any deployment mode names, value paths, or behavioral defaults that diverged between the enterprise and community chart must be reconciled in the umbrella chart's Loki passthrough values. This work must land in the same milestone as the community chart migration. ### Passthrough/Wrapper Pattern Rearchitecture Convert the Loki package from the kpt fork pattern to the BB passthrough/wrapper pattern. This phase begins after the community chart migration is stable on main. Scope includes declaring the community Loki chart as an `alias: upstream` Helm dependency, removing all upstream template files from `chart/templates/` retaining only the `templates/bigbang/` directory, moving all IronBank image overrides under the `upstream:` key in `chart/values.yaml`, and resolving any BB-custom helper logic that cannot be expressed as value overrides. BB-specific templates for dashboards, minio credential generation, and validation remain in `templates/bigbang/`. All bb-common, istio, kyverno, and gluon integrations must continue to function correctly after conversion. A design spike on the minio-tenant storage config helper logic is recommended before implementation begins, as this is the highest-risk area for the conversion. ### Garage Subchart Add [Garage](https://garagehq.deuxfleurs.fr/) as an optional subchart alongside `minio-tenant` for Loki object storage, targeting edge and resource-constrained deployment environments. Garage is a single-binary, S3-compatible object store that requires 1 GB RAM, 16 GB disk, and no operator or sidecar. It replicates data three-way across zones and is designed for unreliable networks and heterogeneous hardware, making it an appropriate alternative to the MinIO operator stack when resource constraints or operational simplicity are priorities. Scope includes integrating Garage as an optional subchart with a disabled-by-default configuration mirroring the structure of the existing `minio-tenant` subchart, extending storage configuration to support Garage as a backend, and documenting when to choose Garage over MinIO. This phase is gated on IronBank image availability for the Garage binary and can be developed in parallel with the passthrough rearchitecture. ## Out of Scope - Migration of existing log data between storage backends (operator responsibility) - Distributed deployment mode support (currently unsupported by BB) - Self-monitoring and grafana-agent-operator (removed upstream in chart 9.0) - Grafana Enterprise Logs support - the OSS-only decision and its rationale are recorded in the Description above; operators requiring GEL should use the BB package template process - Changes to the minio-operator or minio-instance packages ## Duration | Phase | Requirement | Estimate | |-------|-------------|----------| | 1 | Migration vs. Redeploy Decision Record | 0.5 weeks | | 1 | Community Chart Migration | 3 weeks | | 1 | BB Umbrella Chart Alignment | 0.5 weeks (lands with migration) | | 2 | Passthrough Rearchitecture | 3 weeks | | 3 | Garage Subchart | 4 weeks (IronBank gated) | **Total: 9-11 weeks.** Phase 1 is a self-contained deliverable. Phase 2 begins after Phase 1 is stable. Phase 3 can overlap with Phase 2 but ships independently once the IronBank image is available. ## Team Primary: BB Observability team Cross-functional: - BB Platform Engineering (BB umbrella chart alignment) - IronBank team (image request coordination for Garage) ## Epic Team Members - Assignee: TBD - Reviewers: TBD ## Dependencies - BB umbrella chart alignment must land in the same milestone as the community chart migration - Passthrough rearchitecture must begin after the community chart migration is merged to avoid rebasing across an in-flight chart change - Garage subchart is blocked on IronBank image availability for the Garage binary ## Risks **In-place upgrade attempts by operators:** Operators accustomed to `helm upgrade` workflows may attempt an in-place upgrade despite documentation recommending otherwise. Mitigation is to surface the guidance in BB upgrade notices and CHANGELOG entries, not only in package-level documentation. **BB integration regressions:** Migrating to a new upstream chart source creates risk that istio, kyverno, or bb-common integrations behave differently than they did on the enterprise chart. Thorough CI validation across all supported deployment modes and hardening configurations is the primary mitigation. **Umbrella chart timing:** If the BB umbrella chart alignment lands in a different milestone than the community chart migration, there is a window where deprecated configuration is actively in use. The mitigation is enforcing both in the same milestone. **Minio-tenant passthrough compatibility:** The storage config generation logic contains BB-custom code that may not map cleanly to upstream value overrides in the passthrough model. The design spike at the start of Phase 2 is intended to surface this early and determine whether a hybrid approach is needed. **Garage IronBank timeline:** IronBank image requests have unpredictable lead times. The Garage subchart cannot ship without a hardened image regardless of how complete the chart work is. ## Acceptance Criteria - A migration vs. redeploy decision record exists in `docs/DEVELOPMENT_MAINTENANCE.md` with operator instructions for the chosen path - The package source is the `grafana-community/helm-charts` Loki chart with all enterprise and GEL content removed - The package integrates correctly with bb-common, istio, kyverno, and gluon after migration, verified by CI - The BB umbrella chart reflects the community chart's configuration expectations with no deprecated values in active use - The package follows the BB passthrough/wrapper pattern with only `templates/bigbang/` content in `chart/templates/` - Garage can be enabled as an object storage backend with a supported IronBank image reference and passing CI ## Related Issues and/or Epics - [#291](https://repo1.dso.mil/big-bang/product/packages/loki/-/work_items/291) - BB Common egress netpol (intersects with passthrough networking values) - BB umbrella chart issue to be created under the bigbang repository - IronBank image request to be created for Garage ## How This Epic Maps to Big Bang OKRs This epic advances package currency by moving off a deprecated upstream chart source, security posture by reconciling diverged configuration before it causes failures in the field, internal efficiency by reducing per-release template maintenance from 138 files to zero for unchanged upstream releases, and platform breadth by adding a supported storage backend for environments where the MinIO operator is not viable. Reference the [Big Bang Key Results](https://repo1.dso.mil/groups/big-bang/-/epics?label_name%5B%5D=epic%3A%3AKR) for current prioritization context. ## How This Proposed Work Benefits the Enterprise Using Big Bang and/or End User SRE For an enterprise SRE managing a BB cluster, this epic delivers a documented, zero-data-loss procedure for one of the more complex package transitions in the BB ecosystem. It removes enterprise dependencies that create noise in security audits without adding operational value, ensures the full BB hardening stack continues to function correctly after the chart source changes, and for programs operating at the edge or on hardware where the MinIO operator is too resource-intensive, provides a supported, hardened storage backend that requires a single binary and 1 GB of RAM to run. ## How This Benefits the Internal Big Bang Team The passthrough rearchitecture is the most direct internal benefit. The team currently reviews and merges upstream template changes across approximately 138 files on every kpt update for Loki. After conversion, upstream releases that require no BB-specific changes are zero-touch. The community chart's architecture, where individual component templates are thin delegates to shared helper partials, makes this the right moment for the conversion. The umbrella chart alignment work also establishes a clean baseline so that future Loki version updates in the BB umbrella require only value path updates rather than behavioral debugging.