Draft Thoughts on Component Framework
Document Status: DRAFT
No content in this document is final and some sections are still incomplete. Relative level of completion for each top-level section:
- Table of Contents: mostly complete, would need to be updated if sections change.
- Purpose: mostly complete.
- Component Layers: partially complete. Layers have been identified and have some content, but that content is not final. Bulleted content could be reworked into prose to refine the descriptions.
- Component Artifacts (Airgap Support): partially complete. Bulleted content could be reworked into prose to refine and improve the descriptiveness.
- Component Project Structure: mostly complete.
- Crossplane XRD Support: mostly complete.
- Thought Exercise: Deployment Configurations: mostly complete.
- Adapting Current Components: not started. The intent of this section was to identify specific issues in the current components and strategize how to resolve them as part of the move to the new component structure.
Table of Contents
- Purpose
- Component Layers
- Component Dependencies (Airgap Support)
- Crossplane XRD Support
- Thought Exercise: Deployment Configurations
- Adapting Current Components
Purpose
Epic 19 describes the effort to rearchitect how the Edge solution composes its deployment. The current solution architecture is restrictive and defines an Edge deployment as these specific steps in this specific order:
Each of these steps is an Ansible role (linked) and runs a series of tasks to complete part of the deployment. The tasks and roles build on each other such that by the time step 7 is complete, BigBang has been deployed onto the target Edge Hardware device.
There are several drawbacks to this implementation:
- No flexibility in the deployment. This deployment configuration works for a certain category of Edge Hardware, namely, enterprise-grade blade chassises with fast processors, lots of RAM, and petabytes of storage. Other categories of Edge Hardware can't use this deployment configuration, and the restrictive deployment configuration can't support anything else.
- Comingled concerns. Each of the step roles linked is, in theory, an isolated component that would be able to be swapped. However, the implementation of the roles bleeds concerns between roles, meaning none of them are truly isolated and, therefore, could not be swapped
- Hard to extend. As stated above, the roles are not fully isolated and have some bleed over. However, even if they were, there's no clear definition or contract for what each step's purpose is, which makes it difficult to implement an alternative.
The purpose of this document is to propose a new design that addresses these drawbacks and creates a flexible, robust, extensible component framework that can be used to enhance the usability and reliability of the Big Bang Edge solution.
Component Layers
The first step in creating a new component framework is clearly defining the general layers of components that go into any deployment of Big Bang Edge. Each layer definition should include a clear contract - what inputs are required for components in this layer, what do components in this layer generally do, and what are the outputs of components in this layer. These contracts define the minimum set of inputs and outputs; specific components may require additional configuration parameters and outputs to complete its task, which is supported behavior.
The layer structure defined in this section is loosely based on the OSI Model for Cloud published by Cisco. The article talks about a 6-layer standard for building cloud computing platforms. This has a decent degree of overlap with the problem statement Big Bang Edge solves, and so serves as a reasonable starting point for considering layers for the new component framework. However, this design does not adhere exactly to the 6 layers defined by the article.
Layer 1: Infrastructure
- Description: The Infrastructure Layer defines components to be deployed as the foundation of the stack. These typically support modification of bare metal systems through use IPMI and BMC tools.
- Required Inputs:
- Desired IP endpoints of all discrete compute resources
- BMC credentials
- Boot image
- Network configuration
- Required Outputs:
- IP endpoints of all discrete compute resources
- Expected Output State:
- Connection with the target Edge Hardware Device has been tested and confirmed
- The target Edge Hardware Device is in a state where it can have a hypervisor installed. OSes are installed on all compute devices, any necessary preconfiguration has taken place, etc.
Layer 2: Hypervisor
- Description: The Hypervisor layer defines components deployed to support virtualization of the compute resources of the Edge Hardware Device.
- Inputs:
- IP endpoints of all discrete compute resources
- Hypervisor credentials
- Outputs:
- Hypervisor API endpoint
- Hypervisor credentials
- Expected Output State:
- Hypervisor software has been installed on the desired compute resources of the Edge Hardware Device.
- The hypervisor software has been fully configured to allow subsequent components to create virtual machines.
Layer 3: SDDC
- Description: This layer concerns itself with resource pooling, usage tracking, and governance on top of the Hypervisor layer.
- Inputs:
- IP endpoints of all discrete compute resources
- Outputs:
- Expected Output State:
- Internal networking of the Edge Hardware Device is configured.
- Resources such as storage are pooled and available to subsequent components.
Layer 4: Images
- Description: In this layer, the Hypervisor and SDDC are used to deploy particular operating systems and other pre-installed software components. For BBE, this includes installation and configuration of Kubernetes to support Layer 6
- Inputs:
- Outputs:
- Expected Output State:
- Operational OS's have been deployed to the target Edge Hardware Device as configured. This could be VMs or direct install of an OS on the bare metal.
- Images are functional, fully configured (secured) and in a powered-on state.
- A Kubernetes cluster has been created and configured with all nodes available.
Layer 5: Services
- Description: This layer deploys components that provide common services to the later layers. Think databases, load balancers, firewalls, message queues, etc
- Inputs:
- Outputs:
- Expected Output State:
- Common services have either been installed directly on the target Edge Hardware Device OS, or created as virtual machines using the Hypervisor and SDDC components
- Services are properly configured and secured. They are available/reachable within the network of the Edge Hardware Device.
Layer 6: Platform
- Description: The final layer deploys a Kubernetes platform on top of the available Kubernetes cluster.
- Inputs:
- Outputs:
-
Expected Output State:
Layer "Families"
Meta Components
- An object in a layer that doesn't directly deploy something.
- Instead of deployment, meta components do something to another component within the layer.
- Example: a "multi deploy" component, which is a single component that allow configuration of multiple components within it. Facilitates a chain of components in a given layer, which would normally just accept 1 component.
- Another example: a "no-op" component, which is a component that deploys nothing, but will automatically "succeed" and proceed to the next configured component.
- Each layer should have a corresponding MultiXXXComponent and NoOpXXXComponent for use in deployment configurations.
Mapping Current Components
| Current Component | Proposed Layer |
|---|---|
| PXE | Layer 1: Infrastrucure |
| Redfish | Layer 1: Infrastructure |
| Proxmox | Layer 2: Hypervisor |
| S3 (Ceph) | Layer 3: SDDC |
| RKE2 | Layer 4: Images |
| OPNSense | Layer 5: Services |
| BigBang | Layer 6: Platform |
Component Dependencies (Airgap Support)
- File in each component project that contains the manifest of dependencies. File name is consistent across repos to enable automation.
- Common role and task(s) that read this file in to make the dependencies available as variables to Ansible.
- Release tooling that:
- Scrapes the dependency files from all component repos.
- Puts the resulting URLs into the appropriate file in iso-builder so they can be staged.
Example of file:
docker:
rke2-image:https://github.com/rancher/rke2/releases/download/v1.24.0%2Brke2r1/rke2-images-cilium.linux-amd64.tar.zst
rocky_iso_image:https://dl.rockylinux.org/vault/rocky/9.5/isos/x86_64/Rocky-9.5-x86_64-minimal.iso
charts:
crossplane: https://charts.crossplane.io/stableComponent Project Structure
Components in this solution are built on Ansible Operator SDK. In the current architecture, a group of these are all colocated in the bigbang-edge-controller repository. This means any time an Ansible task changes in any component, the whole group needs to be rebuilt, repackaged, and redeployed. Not ideal as the number of components scales.
Instead, each component should be maintained in its own repository. A system similar to the one used for Big Bang packages seems appropriate and scalable. In the Big Bang approach, each package (such as Kyverno, Keycloak, and Istio) has its own repository under the Universe subgroup in repo1. These packages are maintained separately, but common infrastructure builds, packages, and releases these projects in concert with Big Bang.
Specifically, this design proposes:
- Create a new subgroup under the Edge subgroup called
Componentsor similar. - Create a new template repo for Edge Components under the
Componentssubgroup. In this repo, put any common elements that Components are required to have (see the Component Artifacts section for examples), including:- A
dependencies.yamlfile (or similar) that defines all of the component's external dependencies. - A helm chart definition to deploy that component and its resources.
- A
- Take each of the existing components out of
edge-controllerand create repos underComponentsfor each. Use the new template repo as a starting point and move over all existing code for them. - Create a "common" Ansible role project that houses tasks that can be reused by Components. This should be configured to deploy as an Ansible Galaxy role referenced in the
requirements.ymlof Components. One such common task could be a task that reads in the SBOM file for the Component and makes the contents available as Ansible variables. See Component Artifacts for more details. - Create a common pipeline template for Components that can be applied to each repo's CI/CD settings. The pipeline template should publish the Component to the appropriate registry1 location as a Docker image. The Operator SDK supports compiling to Docker images.
- Create or modify release tooling that will publish each of the Component projects on the configured release version of Big Bang Edge. Publishing involves pushing the built image of the Component as well as the associated Helm chart.
- Modify the Helm chart in the edge-operator repo to include all Component helm charts.
Crossplane XRD Support
To facilitate flexible deployment configurations, it's necessary to create a new Crossplane Resource Definition (XRD) that will flexibly deploy a component from each layer. Below is a simplified example of what this could look like, though the full implementation details are out of the scope of this design.
While each layer only supports a single configurable component, a meta component like the examplar MultiComponent can be used to allow multiple components to be configured per layer, if needed. Likewise, if a layer is not required for a deployment configuratoin, the NoOpComponent can be used to pass by that layer while making no changes.
definition.yaml:
---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: flexedgeinstall.platformone.dso.mil
spec:
group: platformone.dso.mil
names:
kind: flexedgeinstall
plural: flexedgeinstalls
versions:
- name: v1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
components:
type: object
properties:
infrastructure:
type: object
properties:
kind:
type: string
description: value for kind of the desired object
config:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- kind
- config
hypervisor:
type: object
properties:
kind:
type: string
description: value for kind of the desired object
config:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- kind
- config
sdcc:
type: object
properties:
kind:
type: string
description: value for kind of the desired object
config:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- kind
- config
images:
type: object
properties:
kind:
type: string
description: value for kind of the desired object
config:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- kind
- config
services:
type: object
properties:
kind:
type: string
description: value for kind of the desired object
config:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- kind
- config
platform:
type: object
properties:
kind:
type: string
description: value for kind of the desired object
config:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- kind
- config
required:
- infrastructure
- hypervisor
- sdcc
- images
- services
- platform
required:
- components
composition.yaml:
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: flexedgeinstall-composition
spec:
compositeTypeRef:
apiVersion: platformone.dso.mil/v1
kind: flexedgeinstall
resources:
- base:
apiVersion: kubernetes.crossplane.io/v1alpha2
kind: Object
spec:
forProvider:
manifest:
apiVersion: infrastructure.platformone.dso.mil/v1
kind: placeholder
metadata:
name: infrastructure-component
spec:
layer: infrastructure
patches:
- fromFieldPath: spec.components.infrastructure.kind
toFieldPath: spec.forProvider.manifest.kind
- fromFieldPath: spec.components.infrastructure.spec
toFieldPath: spec.forProvider.manifest.config
- base:
apiVersion: kubernetes.crossplane.io/v1alpha2
kind: Object
spec:
forProvider:
manifest:
apiVersion: hypervisor.platformone.dso.mil/v1
kind: placeholder
metadata:
name: hypervisor-component
spec:
layer: hypervisor
patches:
- fromFieldPath: spec.components.hypervisor.kind
toFieldPath: spec.forProvider.manifest.kind
- fromFieldPath: spec.components.hypervisor.spec
toFieldPath: spec.forProvider.manifest.config
- base:
apiVersion: kubernetes.crossplane.io/v1alpha2
kind: Object
spec:
forProvider:
manifest:
apiVersion: sdcc.platformone.dso.mil/v1
kind: placeholder
metadata:
name: sdcc-component
spec:
layer: sdcc
patches:
- fromFieldPath: spec.components.sdcc.kind
toFieldPath: spec.forProvider.manifest.kind
- fromFieldPath: spec.components.sdcc.spec
toFieldPath: spec.forProvider.manifest.config
- base:
apiVersion: kubernetes.crossplane.io/v1alpha2
kind: Object
spec:
forProvider:
manifest:
apiVersion: images.platformone.dso.mil/v1
kind: placeholder
metadata:
name: images-component
spec:
layer: images
patches:
- fromFieldPath: spec.components.images.kind
toFieldPath: spec.forProvider.manifest.kind
- fromFieldPath: spec.components.images.spec
toFieldPath: spec.forProvider.manifest.config
- base:
apiVersion: kubernetes.crossplane.io/v1alpha2
kind: Object
spec:
forProvider:
manifest:
apiVersion: services.platformone.dso.mil/v1
kind: placeholder
metadata:
name: services-component
spec:
layer: services
patches:
- fromFieldPath: spec.components.services.kind
toFieldPath: spec.forProvider.manifest.kind
- fromFieldPath: spec.components.services.spec
toFieldPath: spec.forProvider.manifest.config
- base:
apiVersion: kubernetes.crossplane.io/v1alpha2
kind: Object
spec:
forProvider:
manifest:
apiVersion: platform.platformone.dso.mil/v1
kind: placeholder
metadata:
name: platform-component
spec:
layer: platform
patches:
- fromFieldPath: spec.components.platform.kind
toFieldPath: spec.forProvider.manifest.kind
- fromFieldPath: spec.components.platform.spec
toFieldPath: spec.forProvider.manifest.config Thought Exercise: Deployment Configurations
Using the flexible Crossplane Definition, many different deployment configurations could be supported. Below is a thought exercise showing how current or conceived deployment configurations could be accomplished.
HPE EdgeLine Configuration (current default):
---
apiVersion: platformone.dso.mil/v1
kind: flexedgeinstall
metadata:
name: enterprise-datacenter
spec:
components:
infrastructure:
kind: MultiInfrastructureComponent
config:
- kind: PXEInfastructureComponent
config:
...
- kind: RedfishInfrastructureComponent
config:
...
hypervisor:
kind: ProxmoxHypervisorComponent
config:
...
sdcc:
kind: S3SdccComponent
config:
...
images:
kind: RockyRKE2ProxmoxImageComponent
config:
...
services:
kind: OPNSenseServiceComponent
config:
...
platform:
kind: BigBangPlatformComponent
config:
...Slimmed down direct install to SimpluNUC EE-1000 hardware as envisioned by epic &46:
---
apiVersion: platformone.dso.mil/v1
kind: flexedgeinstall
metadata:
name: mini-node
spec:
components:
infrastructure:
kind: MultiInfrastructureComponent
config:
- kind: UARTInfastructureComponent
config:
...
- kind: PXEInfastructureComponent
config:
...
- kind: RedfishInfastructureComponent
config:
...
hypervisor:
kind: NoOpHypervisorComponent
config:
...
sdcc:
kind: NoOpSdccComponent
config:
...
images:
kind: UbuntuRKE2MetalImageComponent
config:
...
services:
kind: NoOpServicesComponent
config:
...
platform:
kind: BigBangPlatformComponent
config:
...