add unified package configuration for 3.x

General MR

Summary

This merge request lays the foundation for the Big Bang 4.x unified package configuration model while preserving existing Big Bang 3.x behavior.

The long-term configuration contract places built-in and user-supplied packages under:

packages:
  <package-name>:
    ...

Because canonical built-in names overlap with the existing custom-package namespace, the new behavior is explicitly enabled in 3.x through:

packageConfiguration:
  version: v1

Existing installations that do not set this version retain the current custom-package interpretation of every entry under packages.

What changed

Package metadata and generation

• Added chart/package-metadata.yaml as the authoritative catalog for the 44 built-in packages.
• Recorded each package’s canonical name, legacy path, category, template directory, display name, and documentation location.
• Added metadata-driven schema generation and migration-map generation.
• Added bidirectional catalog validation:
  • Every catalog entry must reference an existing package template directory.
  • Every integrated package template directory containing helmrelease.yaml must be represented in the catalog.
• Added identifier validation requireing camelCase package keys and kebab-case template directory names.
• Metadata identifiers are validated before they are used as filesystem paths or emitted into generated Bash arrays.
• Added BATS coverage for uncatalogued integrated packages and invalid metadata identifiers.
• Added checks ensuring generated schemas and migration mappings remain current.
• Integrated the generation check with repository test and hook workflows.

Canonical package normalization

• Added a compatibility normalizer that maps canonical built-in values onto the legacy paths consumed by existing templates.
• Canonical values recursively override legacy values when both are supplied.
• Explicitly supplied and resolved canonical values remain available under .Values.packages, allowing tpl expressions to reference the new path.
• Legacy-only built-ins are not automatically copied into packages; populating all built-in defaults remains 4.x work.
• Generic package, wrapper, namespace, secret, GitRepository, HelmRelease, and Kustomization renderers now consume a filtered _customPackages map.
• This prevents canonical built-ins from also rendering as generic custom packages.

Backward compatibility and collision handling

• Added the opt-in packageConfiguration.version: v1 contract.
• When the version is unset, existing 3.x custom-package behavior remains unchanged.
• Added validation for package names that case-fold or normalize to built-in package identities or resource names.
• Added validation for collisions between custom packages that normalize to the same Kubernetes resource identity.
• Added clear errors instead of silently reinterpreting or duplicating packages.

Schema validation

• Added generated metadata-driven partial schemas for canonical built-in packages.
• Partial schemas validate supplied canonical fields without requiring a complete package definition.
• Open-ended child-chart values remain supported.
• Schema selection is conditional:
  • Unversioned configurations use the existing custom-package schema.
  • v1 configurations use canonical built-in schemas while still allowing non-conflicting custom packages.
• Unsupported package configuration versions are rejected.

The large schema change is generated output rather than manually maintained validation. It can be simplified in 4.x when the legacy conditional contract is removed.

Migration tooling

Added scripts/migrate-values-3-to-4.sh to migrate legacy package configuration safely.

The utility:

• Moves core and add-on package configuration to packages.<name>.
• Sets packageConfiguration.version: v1.
• Preserves canonical-over-legacy precedence.
• Accepts multiple values files in Helm order and produces one consolidated migrated document.
• Recursively merges maps and replaces arrays according to later-file precedence.
• Migrates deprecated addons.mattermostoperator below addons.mattermostOperator and packages.mattermostOperator in precedence.
• Refuses to reinterpret an unversioned custom package whose name exactly matches a built-in.
• Rejects SOPS-encrypted input with decrypt/migrate/re-encrypt guidance.
• Rejects unsupported multi-document YAML.
• Rejects YAML anchors and aliases rather than rewriting them unpredictably.
• Prevents output symlinks and hardlinks from overwriting an input.
• Supports safe single-file in-place migration with backup creation.
• Is idempotent.

Documentation

• Added an ADR describing the unified package contract and phased rollout.
• Added a Big Bang 4.x package-values migration guide.
• Updated package-management and configuration documentation.
• Clarified that the migration guide and scripts/migrate-values-3-to-4.sh cover package-path migration only. They preserve but do not rewrite other deprecated Big Bang settings or child-chart values, including legacy hostname, SSO, Istio hardening, and bb-common compatibility values.
• Documented the durable packageConfiguration.version: v1 contract, precedence behavior, migration workflow, safety restrictions, and eventual 4.x cleanup.

Why

This provides users with a safe migration path before the 4.x breaking change while establishing the package metadata, validation, rendering, and automation foundations needed for the final unified model.

The approach deliberately avoids moving all built-in defaults under packages during 3.x. That larger breaking change remains reserved for 4.x, when the legacy paths, compatibility normalizer, and conditional schema branches can be removed. The packageConfiguration.version: v1 contract discriminator remains supported in 4.x.

Backward compatibility

Existing 3.x installations remain on the legacy/custom-package contract unless they explicitly enable:

packageConfiguration: version: v1

The migration utility selects the packageConfiguration.version: v1 contract intentionally...

No package versions are changed by this merge request.

What should wait for 4.x

Do not move defaults or rewrite all package templates to read canonical paths in this minor release. The compatibility normalizer is an appropriate temporary adapter. In 4.x, make one clean cut:

• Move built-in defaults under packages.
• Make templates consume only canonical paths.
• Remove legacy schema paths and alias mutation.
• Retain the metadata catalog as the authoritative package inventory.

Relevant logs/screenshots

n/a

Linked Issue

issue

Upgrade Notices

Big Bang 3.x now supports the unified package configuration planned for Big Bang 4.x. Built-in and custom packages can be configured consistently under packages.<name> by selecting Configuration Version 1:

packageConfiguration:
  version: v1

Existing top-level and addons.<name> configuration remains supported throughout Big Bang 3.x, so no immediate migration is required. When Configuration Version 1 is selected and both legacy and canonical paths configure the same package, packages.<name> takes precedence.

packageConfiguration:
  version: v1

packages:
  monitoring:
    enabled: true
  gitlab:
    enabled: true

Users are encouraged to migrate and validate their configuration while still running Big Bang 3.x. An automated migration utility is provided:

scripts/migrate-values-3-to-4.sh values.yaml > values-4.x.yaml

Review and deploy the migrated values with Big Bang 3.x before upgrading to 4.x. The migration utility adds packageConfiguration.version: v1; retain this setting when upgrading. Configuration Version 1 is the durable unified-package contract discriminator and becomes the default package contract in Big Bang 4.x.

Big Bang 4.x will remove the legacy top-level package paths, the addons package mapping, and the associated 3.x compatibility layer.

Existing unversioned custom packages whose names conflict with built-in packages require manual resolution before migration. The migration utility detects these collisions and stops rather than reinterpreting the package.

For complete instructions, examples, limitations, and collision-handling guidance, see the Big Bang 4.0 package values migration guide.