UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • big-bang/bigbang
  • joshwolf/umbrella
  • 90-cos/iac/bigbang
  • cbrechbuhl/bigbang
  • runyontr/bigbang-core
  • snekcode/bigbang
  • michael.mendez/bigbang
  • daniel.dides/bigbang
  • ryan.j.garcia/rjgbigbang
  • nicole.dupree/bigbang
10 results
Show changes
Commits on Source (54)
Hide whitespace changes
Inline Side-by-side
Showing
with 233 additions and 349 deletions
.gitlab-ci.yml
View file @ 256d554d
......@@ -64,9 +64,16 @@ pre vars:
reports:
dotenv: variables.env
script:
# Create the TF_VAR_env variable
- echo "TF_VAR_env=$(echo $CI_COMMIT_REF_SLUG | cut -c 1-7)-$(echo $CI_COMMIT_SHA | cut -c 1-7)" >> variables.env
- cat variables.env
# Create the TF_VAR_env variable
- echo "TF_VAR_env=$(echo $CI_COMMIT_REF_SLUG | cut -c 1-7)-$(echo $CI_COMMIT_SHA | cut -c 1-7)" >> variables.env
- cat variables.env
retry:
max: 2
when:
- unknown_failure
- stuck_or_timeout_failure
- runner_system_failure
#-----------------------------------------------------------------------------------------------------------------------
......@@ -136,6 +143,12 @@ clean install:
when: always
allow_failure:
exit_codes: 123
retry:
max: 2
when:
- unknown_failure
- stuck_or_timeout_failure
- runner_system_failure
upgrade:
stage: smoke tests
......@@ -176,6 +189,12 @@ upgrade:
when: always
allow_failure:
exit_codes: 123
retry:
max: 2
when:
- unknown_failure
- stuck_or_timeout_failure
- runner_system_failure
#-----------------------------------------------------------------------------------------------------------------------
# Rules for execution of AWS based K3S cluster deployment: Infrastructure jobs
......@@ -282,7 +301,6 @@ aws/rke2/bigbang up:
- cp ${CI_PROJECT_DIR}/rke2.yaml ~/.kube/config
# Deploy a default storage class for aws
- kubectl apply -f ${CI_PROJECT_DIR}/.gitlab-ci/jobs/rke2/dependencies/k8s-resources/aws/default-ebs-sc.yaml
script:
- *deploy_bigbang
environment:
......@@ -352,6 +370,12 @@ aws/rke2/bigbang down:
- sleep 180
environment:
name: review/aws-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}
retry:
max: 2
when:
- unknown_failure
- stuck_or_timeout_failure
- runner_system_failure
# Destroy RKE2 cluster on AWS
aws/rke2/cluster down:
......@@ -435,6 +459,12 @@ package:
aws s3 sync --quiet release/ s3://umbrella-bigbang-releases/umbrella/${CI_COMMIT_TAG}
fi
after_script: []
retry:
max: 2
when:
- unknown_failure
- stuck_or_timeout_failure
- runner_system_failure
release:
stage: release
......@@ -464,5 +494,11 @@ release:
--assets-link "{\"name\":\"${IMAGE_PKG}\",\"url\":\"${RELEASE_ENDPOINT}/${IMAGE_PKG}\"}" \
--assets-link "{\"name\":\"${REPOS_PKG}\",\"url\":\"${RELEASE_ENDPOINT}/${REPOS_PKG}\"}"
fi
retry:
max: 2
when:
- unknown_failure
- stuck_or_timeout_failure
- runner_system_failure
#-----------------------------------------------------------------------------------------------------------------------
CODEOWNERS
View file @ 256d554d
......@@ -79,9 +79,9 @@ chart/values.yaml @lynnstill @ryan.j.garcia @michaelmartin
chart/templates/monitoring @lynnstill @ryan.j.garcia @michaelmartin
^[Twistlock]
chart/Chart.yaml @thomas.burton @ryan.j.garcia @runyontr @joshwolf
chart/values.yaml @thomas.burton @ryan.j.garcia @runyontr @joshwolf
chart/templates/twistlock @thomas.burton @ryan.j.garcia @runyontr @joshwolf
chart/Chart.yaml @thomas.burton @ryan.j.garcia @runyontr @micah.nagel
chart/values.yaml @thomas.burton @ryan.j.garcia @runyontr @micah.nagel
chart/templates/twistlock @thomas.burton @ryan.j.garcia @runyontr @micah.nagel
^[Sonarqube]
chart/Chart.yaml @kevin.wilder @lynnstill @brandencobb
......
chart/templates/istio/controlplane/values.yaml
View file @ 256d554d
......@@ -47,6 +47,8 @@ gateways:
{{ $name | nindent 2 }}:
selector:
app: {{ $values.ingressGateway }}
autoHttpRedirect:
enabled: {{ dig "autoHttpRedirect" "enabled" "true" $values }}
servers:
- hosts:
{{ tpl ($values.hosts | default (list) | toYaml) $ | nindent 8 }}
......
chart/templates/jaeger/values.yaml
View file @ 256d554d
......@@ -21,6 +21,10 @@ monitoring:
enabled: {{ .Values.monitoring.enabled }}
elasticsearch:
enabled: {{ .Values.logging.enabled }}
sso:
enabled: {{ .Values.jaeger.sso.enabled }}
{{- if .Values.jaeger.sso.enabled }}
jaeger:
spec:
......@@ -42,4 +46,4 @@ networkPolicies:
{{- $gateway := default "public" .Values.jaeger.ingress.gateway }}
{{- $default := dict "app" (dig "gateways" $gateway "ingressGateway" nil .Values.istio) "istio" nil }}
{{- toYaml (dig "values" "gateways" $gateway "selector" $default .Values.istio) | nindent 4 }}
{{- end -}}
\ No newline at end of file
{{- end -}}
chart/templates/sonarqube/namespace.yaml
View file @ 256d554d
......@@ -7,5 +7,5 @@ metadata:
app.kubernetes.io/name: sonarqube
app.kubernetes.io/component: "developer-tools"
{{- include "commonLabels" . | nindent 4}}
istio-injection: disabled
istio-injection: {{ dig "istio" "injection" "enabled" .Values.addons.sonarqube }}
{{- end }}
chart/templates/sonarqube/values.yaml
View file @ 256d554d
......@@ -13,6 +13,7 @@ istio:
sonarqube:
gateways:
- istio-system/{{ default "public" .Values.addons.sonarqube.ingress.gateway }}
injection: {{ dig "istio" "injection" "enabled" .Values.addons.sonarqube }}
monitoring:
enabled: {{ .Values.monitoring.enabled }}
......
chart/values.yaml
View file @ 256d554d
......@@ -118,7 +118,7 @@ istio:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-controlplane.git
path: "./chart"
tag: "1.11.2-bb.0"
tag: "1.11.2-bb.2"
# Ingress gateways are created based on the key name. Adding more keys will add ingress gateways.
# Ingress gateways are setup in a Horizontal Pod Autoscaler with 1 to 5 replicas
......@@ -148,6 +148,9 @@ istio:
ingressGateway: "public-ingressgateway"
hosts:
- "*.{{ .Values.domain }}"
# -- Controls default HTTP/8080 server entry with HTTP to HTTPS Redirect.
autoHttpRedirect:
enabled: true
tls:
key: ""
cert: ""
......@@ -155,6 +158,9 @@ istio:
# ingressGateway: "private-ingressgateway"
# hosts:
# - "*.{{ .Values.domain }}"
# # -- Controls default HTTP/8080 server entry with HTTP to HTTPS Redirect.
# autoHttpRedirect:
# enabled: true
# tls:
# key: ""
# cert: ""
......@@ -162,6 +168,9 @@ istio:
# ingressGateway: "passthrough-ingressgateway"
# hosts:
# - "*.{{ .Values.domain }}"
# # -- Controls default HTTP/8080 server entry with HTTP to HTTPS Redirect.
# autoHttpRedirect:
# enabled: true
# tls:
# mode: "PASSTHROUGH"
......@@ -197,7 +206,7 @@ jaeger:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/core/jaeger.git
path: "./chart"
tag: "2.23.0-bb.3"
tag: "2.23.0-bb.4"
# -- Flux reconciliation overrides specifically for the Jaeger Package
flux:
......@@ -232,7 +241,7 @@ kiali:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/core/kiali.git
path: "./chart"
tag: "1.39.0-bb.3"
tag: "1.40.1-bb.0"
# -- Flux reconciliation overrides specifically for the Kiali Package
flux: {}
......@@ -314,7 +323,7 @@ logging:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana.git
path: "./chart"
tag: "0.1.21-bb.1"
tag: "0.1.21-bb.2"
# -- Flux reconciliation overrides specifically for the Logging (EFK) Package
flux:
......@@ -450,7 +459,7 @@ twistlock:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/twistlock.git
path: "./chart"
tag: "0.0.9-bb.0"
tag: "0.0.9-bb.1"
# -- Flux reconciliation overrides specifically for the Twistlock Package
flux: {}
......@@ -560,7 +569,7 @@ addons:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/application-utilities/minio.git
path: "./chart"
tag: "4.2.3-bb.3"
tag: "4.2.3-bb.5"
# -- Flux reconciliation overrides specifically for the Minio Package
flux: {}
......@@ -593,7 +602,7 @@ addons:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab.git
path: "./chart"
tag: "5.3.1-bb.0"
tag: "5.3.1-bb.2"
# -- Flux reconciliation overrides specifically for the Gitlab Package
flux: {}
......@@ -663,12 +672,12 @@ addons:
postRenderers: []
gitlabRunner:
# -- Toggle deployment of Gitlab Runner.
# -- Toggle deployment of Gitlab Runner
enabled: false
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab-runner.git
path: "./chart"
tag: "0.32.0-bb.1"
tag: "0.33.1-bb.2"
# -- Flux reconciliation overrides specifically for the Gitlab Runner Package
flux: {}
......@@ -685,7 +694,7 @@ addons:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/nexus.git
path: "./chart"
tag: "34.0.0-bb.0"
tag: "34.1.0-bb.2"
# -- Base64 encoded license file.
license_key: ""
......@@ -744,7 +753,7 @@ addons:
git:
repo: https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/sonarqube.git
path: "./chart"
tag: "9.6.3-bb.2"
tag: "9.6.3-bb.8"
# -- Flux reconciliation overrides specifically for the Sonarqube Package
flux: {}
......
charter/BigBangPackages.md
View file @ 256d554d
......@@ -25,7 +25,6 @@ graph TB
Thanos
end
ServiceMesh
ArgoCD
ClusterAuditor --> LoggingECK
ClusterAuditor --> OPA(Policy Enforcement)
......@@ -35,8 +34,6 @@ graph TB
Postgres
MinIO(S3 Compatible Storage)
Redis
MySQL
MongoDB
end
subgraph "Security"
......@@ -54,8 +51,6 @@ graph TB
end
subgraph "Collaboration Tools"
Jira --> Postgres
Confluence --> Postgres
MatterMost --> MinIO
end
......@@ -82,7 +77,6 @@ graph TB
Thanos
end
ServiceMesh
ArgoCD
Twistlock
ClusterAuditor --> LoggingECK
......@@ -90,27 +84,6 @@ graph TB
end
```
### ArgoCD
Product:
* [ArgoCD](https://argoproj.github.io/argo-cd/)
Repository:
* [ArgoCD Repo](https://repo1.dso.mil/platform-one/big-bang/apps/core/argocd)
Dependency: None
Owners:
* @joshwolf - Rancher Federal
* @karchaf
Understudy:
* @kavitha
### Service Mesh
Current implementation of Service Mesh is provided by Istio. Service Mesh should be the first Package deployed to ensure other applications are operating with visibility and security.
......@@ -127,15 +100,7 @@ Repository:
Dependency: None
Owners:
* @runyontr - Runyon Solutions
* @nick_tetrate - Tetrate
Understudy:
* Chris McGrath
* @kavitha
* @kenna81
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-operator/-/blob/main/CODEOWNERS)
### Auth Service
......@@ -147,17 +112,12 @@ Product:
Repository:
* [authservice](https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/authservice)
* [authservice](https://repo1.dso.mil/platform-one/big-bang/apps/core/authservice)
Dependency: None
Owners:
* @runyontr - Runyon Solutions
* @nick_tetrate - Tetrate
* @adam.toy - Rancher Federal
Understudy:
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/authservice/-/blob/main/CODEOWNERS)
### Logging
......@@ -182,13 +142,9 @@ Dependencies:
* RWO StorageClass
Owners:
* @kavitha
* @ryan.j.garcia
Understudy:
* @evan.rush
* [Elasticsearch-kibana CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana/-/blob/main/CODEOWNERS)
* [Fluentbit CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit/-/blob/main/CODEOWNERS)
* [Eck-operator CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/eck-operator/-/blob/main/CODEOWNERS)
### Policy Enforcement
......@@ -207,13 +163,7 @@ Dependencies: None
Owners:
* @runyontr - Runyon Solutions
* @karchaf - Cloud Fit Software
Understudy
* @agudem
* @kavitha
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/policy/-/blob/main/CODEOWNERS)
### Monitoring
......@@ -233,8 +183,7 @@ Dependencies: None
Owners:
* @lynnStill
* @ryan.j.garcia
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring/-/blob/main/CODEOWNERS)
### Cluster Auditor
......@@ -251,12 +200,7 @@ Dependencies:
Owners:
* @runyontr - Runyon Solutions
* @thomas.burton - iSenpai
Understudy:
* @kenna81
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/cluster-auditor/-/blob/main/CODEOWNERS)
Repository:
......@@ -278,8 +222,7 @@ Dependencies:
Owners:
* @runyontr - Runyon Solutions
* @thomas.burton - iSenpai
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/twistlock/-/blob/main/CODEOWNERS)
## Addons
......@@ -319,14 +262,7 @@ Dependencies:
Owners:
* @megamind
* @kevin.wilder
* @michaelmcleroy
Understudy:
* @agudem
* @kenna81
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/keycloak/-/blob/main/CODEOWNERS)
#### Anchore Enterprise
......@@ -342,8 +278,7 @@ Dependencies:
Owners:
* @thomas.burton - iSenpai
* @james.peterson - Anchore
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/anchore-enterprise/-/blob/main/CODEOWNERS)
### Developer Tools
......@@ -389,8 +324,7 @@ Dependencies:
Owners:
* @ryan.j.garcia
* @LynnStill
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab/-/blob/main/CODEOWNERS)
#### GitLab Runners
......@@ -410,12 +344,7 @@ Dependencies:
Owners:
* @ryan.j.garcia
* @LynnStill
Understudies
* @kevin.wilder
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab-runner/-/blob/main/CODEOWNERS)
#### Sonarqube
......@@ -436,27 +365,7 @@ Dependencies:
Owners:
* @kevin.wilder
* @LynnStill
#### Fortify
Fortify provides code
Product:
*
Repository:
* [Fortify Repo](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/fortify)
Dependencies:
Owners:
* @kevin.wilder
* @LynnStill
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/sonarqube/-/blob/main/CODEOWNERS)
#### Nexus
......@@ -471,15 +380,13 @@ Product:
Repository:
* [Nexus](https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/nexus)
* [Nexus](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/nexus)
Dependencies:
Owners:
* @kevin.wilder
* @ariel.shnitzer
* @grant.duncklee
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/nexus/-/blob/main/CODEOWNERS)
### Collaboration Tools
......@@ -500,7 +407,7 @@ graph TB
```
#### Confluence
<!-- #### Confluence
Confluence provides a centralized workspace for collaborating on documentation
......@@ -542,7 +449,7 @@ Dependencies:
Owners:
* @matt.kaiser
* @branden.cobb
* @branden.cobb -->
#### Mattermost
......@@ -562,8 +469,7 @@ Dependencies:
Owners:
* @ryan.j.garcia
* @kevin.wilder
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/collaboration-tools/mattermost/-/blob/main/CODEOWNERS)
### Package Utilities
......@@ -606,14 +512,15 @@ Product:
* [MinIO](https://min.io/)
Repository: TBD
Repository:
* [Minio Package](https://repo1.dso.mil/platform-one/big-bang/apps/application-utilities/minio/)
Dependencies: None
Owners:
* @kevin.wilder - Dark Wolf Solutions
* @branden.cobb
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/application-utilities/minio/-/blob/main/CODEOWNERS)
#### MySQL
......@@ -657,13 +564,28 @@ Repository:
Owners:
* @runyontr - Runyon Solutions
* @still - Parsons
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/redis/-/blob/main/CODEOWNERS)
### Cluster Utilities
Packages that provider cluster level utility, such as RWX storage or generic backup capabilities.
#### ArgoCD
Product:
* [ArgoCD](https://argoproj.github.io/argo-cd/)
Repository:
* [ArgoCD Repo](https://repo1.dso.mil/platform-one/big-bang/apps/core/argocd)
Dependency: None
Owners:
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/core/argocd/-/blob/main/CODEOWNERS)
#### Velero
Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes
......@@ -676,13 +598,16 @@ Repository:
Owners:
* @tunde - Oteemo
* @adam.toy - Rancher Federal
* [CODEOWNERS](https://repo1.dso.mil/platform-one/big-bang/apps/cluster-utilities/velero/-/blob/main/CODEOWNERS)
### BB Technical Oversight Committee (BB TOC)
### Sandbox
[Process](https://repo1.dso.mil/platform-one/bbtoc/-/tree/master/process)
The [Sandbox](https://repo1.dso.mil/platform-one/big-bang/apps/sandbox) is an area for packages that are currently being or will be worked that do not yet meet the requirements of a supported package. Due to the fluidity of sandbox apps, they are not tracked in the charter.
#### BB TOC Repos
[Graduated](https://repo1.dso.mil/platform-one/big-bang/apps/graduated)
Note, this is _not_ a place where packages go to die. If a package is abandoned for whatever reason it will be archived.
[Incubating](https://repo1.dso.mil/platform-one/big-bang/apps/incubating)
To graduate from a sandbox package, it must meet the requirements outlined in this charter.
[Sandbox](https://repo1.dso.mil/platform-one/big-bang/apps/sandbox)
\ No newline at end of file
charter/NewPackageRequests.md
View file @ 256d554d
......@@ -2,9 +2,7 @@
This is the process for adding a new package into Big Bang
## Out-of-Tree / 3rd Party Packages
### Submit New Big Bang Package Proposal to the BB Technical Oversite Committee
## Submit New Big Bang Package Proposal to the BB Technical Oversite Committee
[BB TOC New Package Proposal](https://repo1.dso.mil/platform-one/p1toc/-/issues/new?issue%5Bassignee_id%5D=&issue%5Bmilestone_id%5D=)
......@@ -12,8 +10,4 @@ A shepherd will be assigned to the project to create a repo in the [BB sandbox](
### Process
Out-of-Tree packages packages will follow the [BBTOC process](https://repo1.dso.mil/platform-one/bbtoc/-/tree/master/process) from Sandbox -> Incubating -> Graduated
## In-Tree / Big Bang Maintained Package Process
In order for a package to become an "In-Tree" package (supported by Platform One), it must meet all of the requirements of a BB TOC graduated package and have an [issue](https://repo1.dso.mil/groups/platform-one/big-bang/apps/third-party/-/issues) opened to 'Recommend Package for "In-Tree" Support'. The issue will be processed through the Platform One Jedi Order and Rebel Alliance councils for a governement decision to be added as officially supported / "in-tree" add-on.
New packages packages will follow the [BBTOC process](https://repo1.dso.mil/platform-one/bbtoc/-/tree/master/process) from Sandbox -> Incubating -> Graduated
charter/PackageRequirements.md
View file @ 256d554d
......@@ -24,17 +24,9 @@ Each package will work with any cluster under the following criteria.
## PR-X. Iron Bank Images
Every Big Bang Package shall be configured to use Iron Bank images. The images used from Iron Bank __must__ be _fully_ approved and _functional_ to be in compliance with the Big Bang baseline security posture.
Big Bang Package shall be configured to use Iron Bank images. The images used from Iron Bank __must__ be _fully_ approved and _functional_ to be in compliance with the Big Bang baseline security posture.
Once this prerequisite is met, a package is eligible for inclusion within BigBang in accordance with [New Package Requests](NewPackageRequests.md).
### Out-of-Tree Packages
[Out-of-Tree Packages](https://repo1.dso.mil/platform-one/big-bang/apps/third-party) are third party packages that adhere to all the BigBang package standards. These packages are predominantly community-maintained packages; however, some packages may be jointly maintained by BigBang and community as indicated by the codeowners.
### In-Tree Packages
[In-Tree Packages](https://repo1.dso.mil/platform-one/big-bang/apps) are Platform One developer-supported Big Bang Core & add-ons that adhere to all the BigBang package standards. These packages have been adopted as an official Big Bang offering for key customers. As such, they are supported, updated, and maintained by team members of BigBang and are labeled with the "BigBang Supported" badge on the repository's `README.md` page, which indicates active support. That being said, BigBang reserves the right to deprecate support for these packages.
Please see [New Package Requests](NewPackageRequests.md) and the [BBTOC process](https://repo1.dso.mil/platform-one/bbtoc/-/tree/master/process) for additional pre-requisites.
## PR-X. Packages are Helm Charts
......@@ -83,7 +75,7 @@ include:
file: '/templates/package-tests.yaml'
```
## PR-X. Dependencies must be Big Bang Package
## PR-X. Dependencies must be a Big Bang Package
If a Package has a dependency on another Package to function, the dependency shall also be a Big Bang Package
......
docs/developer/development-environment.md
View file @ 256d554d
......@@ -20,6 +20,9 @@ This page contains the manual steps to create your k3d dev environment. There is
- [Helm](https://helm.sh/docs/intro/install/)
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
- [kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize/)
> For additional installtion details, see [Software Installation and Verification Commands to run from Bash](https://repo1.dso.mil/platform-one/onboarding/big-bang/engineering-cohort/-/blob/master/lab_guides/01-Preflight-Access-Checks/A-software-check.md)
## Manual Creation of a Development Environment
......
docs/developer/package-integration.md
View file @ 256d554d
# Integrate a Package with Bigbang Helm Chart
[[_TOC_]]
1. Make a branch from the BigBang chart repository master branch. You can automatically create a branch from the Repo1 Gitlab issue. Or, in some cases you might manually create the branch. You should name the branch with your issue number. If your issue number is 9999 then your branch name can be "9999-my-description". It is best practice to make branch names short and simple.
1. Create a directory for your package at `chart/templates/<your-package-name>`
1. Inside this folder will be various helm template files. The rule is one document per yaml file. You can copy one of the other package folders and tweak the code for your package. Gitlab is a good example to reference because it is one of the more complicated Packages. Note that the Istio VirtualService comes from the Package and is not created in the BigBang chart. The purpose of these helm template files is to create an easy-to-use spec for deploying supported applications. Reasonable and safe defaults are provided and any needed secrets are auto-created. We accept the trade off of easy deployment for complicated template code. More details are in the following steps.
```shell
gitrepository.yaml # Flux GitRepository. Is configured by BigBang chart values.
helmrelease.yaml # Flux HelmRelease. Is configured by BigBang chart values.
namespace.yaml # Contains the namespace and any needed secrets
secret-*.yaml # various template files that create any needed k8s secrets
values.yaml # Implements all the BigBang customizations of the package and passthrough for values.
```
1. More details about values.yaml: Code reasonable and safe defaults but prioritize any user defined passthrough values wherever this makes sense. Avoid duplicating tags that are provided in the upstream chart values. Instead code reasonable defaults in the values.yaml template. The following is an example from Gitlab that handles SSO config. The code uses Package chart passthrough values if the user has entered them but otherwise defaults to the BigBang chart values or the Helm default values. Notice that the secret is not handled this way. The assumption is that if the user has enabled the BigBang SSO feature the secret will be auto created. In this case the user should not be overriding the secret. If the user wants to create their own secret they should not be enabling the BigBang SSO feature.
Note that helm does not handle any missing parent tags in the yaml tree. The 'if' statement and 'default' method throw 'nil' errors when parent tags are missing. The work-around is to inspect each level of the tree and assign an empty 'dict' if the value does not exist. Then you will be able to use 'hasKey' in your 'if' statements as shown below in this example from Gitlab. Having described all this, you should understand that coding conditional values is optional. The passthrough values will take priority regardless. But the overridden values will not show up in the deployed flux HelmRelease object if you don't code the conditional values. The value overrides will be obscured in the Package values secret. The only way to confirm that the overrides have been applied is to use `helm get values <releasename> -n bigbang` command on the deployed helm release. When the passthrough values show up in the HelmRelease object the Package configuration is much easier to see and verify. Use your own judgement on when to code conditional values.
```yaml
global:
{{- if or .Values.addons.gitlab.sso.enabled .Values.addons.gitlab.objectStorage.endpoint }}
appConfig:
{{- end }}
{{- if .Values.addons.gitlab.sso.enabled }}
omniauth:
enabled: true
{{- $global := .Values.addons.gitlab.values.global | default dict }}
{{- $appConfig := $global.appConfig | default dict }}
{{- $omniauth := $appConfig.omniauth | default dict }}
{{- if hasKey $omniauth "allowSingleSignOn" }}
allowSingleSignOn: {{ .Values.addons.gitlab.values.global.appConfig.omniauth.allowSingleSignOn }}
{{- else }}
allowSingleSignOn: ['openid_connect']
{{- end }}
{{- if hasKey $omniauth "blockAutoCreatedUsers" }}
blockAutoCreatedUsers: {{ .Values.addons.gitlab.values.global.appConfig.omniauth.blockAutoCreatedUsers }}
{{- else }}
blockAutoCreatedUsers: false
{{- end }}
providers:
- secret: gitlab-sso-provider
key: gitlab-sso.json
{{- end }}
```
1. More details about secret-*.yaml: The secret template is where the code for secrets go. Typically you will see secrets for imagePullSecret, sso, database, and possibly object storage. These secrets are a BigBang chart enhancement. They are created conditionally based on what the user enables in the config.
1. Edit the chart/templates/values.yaml. Add your Package to the list of Packages. Just copy one of the others and change the name. This supports adding chart values from a secret. Pay attention to whether this is a core Package or an add-on package, the toYaml values are different for add-ons. This template allows a Package to add chart values that need to be encrypted in a secret.
1. Edit the `chart/values.yaml`. Add your Package to the bottom of the core section if a core package or addons section if an add-on. You can copy from one of the other packages and modify appropriately. Some possible tags underneath your package are [enabled, git, sso, database, objectstorage]. Avoid duplicating value tags from the upstream chart in the BigBang chart. The goal is not to cover every edge case. Instead code reasonable defaults in the helmrelease template and allow customer to override values in addons.`<packageName>.values`
```yaml
addons:
mypackage:
enabled: false # default to false
git:
repo: https://repo1.dsop.io/platform-one/big-bang/apps/developer-tools/mypackage.git
path: "./chart"
tag: "1.2.3-bb.0"
sso:
enabled: false # default to false
client_id: ""
database:
host: ""
port: ""
username: ""
database: ""
password: {} # unencoded stringData
objectstorage:
type: s3 # supported types are "s3" or "minio"
endpoint: "" # ignored if type is "s3". used only for minio. example " http://minio.minio.svc.cluster.local:9000"
host: s3.amazonaws.com # used for gitlab backup storage
region: us-west-1
accessKey: ""
accessSecret: ""
bucketPrefix: "" # optional. example: dev-
values: {}
```
1. Edit tests/ci/k3d/values.yaml. These are the settings that the CI pipeline uses to run a deployment test. Set your Package to be enabled and add any other necessary values. Where possible reduce the number of replicas to a minimum to reduce strain on the CI infrastructure. When you commit your code the pipeline will run. You can view the pipeline in the Repo1 Gitlab console. Fix any errors in the pipeline output. The pipeline automatically runs a "smoke" test. It deploys bigbang on a k3d cluster using the test values file.
1. Add your packages name to the ORDERED_HELMRELEASES list in scripts/deploy/02_wait_for_helmreleases.sh.
1. Create an overrrides directory as a sibling directory next to the bigbang code directory. Put your override yaml files in this directory. The reason we do this is to avoid modifying the bigbang values.yaml that is under source control. You could accidentally commit it with your secrets. Avoid that mistake and create a local overrides directory. One option is to copy the tests/ci/k3d/values.yaml to make the override-values.yaml and make modifications. The file structure is like this:
```text
├── bigbang/
└── overrides/
├── override-values.yaml
├── registry-values.yaml
└── any-other-values.yaml
```
Make the registry-values yaml like this:
```yaml
registryCredentials:
- registry: registry1.dso.mil
username: your-name
password: your-pull-token
email: xxx@xxx.xxx
```
You will use these files as arguments in your helm commands.
1. Verify your Package works when deployed through bigbang. See the instructions below in the ```BigBang Development and Testing Cycle``` for the manual ```imperative``` way to deploy with helm upgrade commands. While testing you should use your package git branch instead of a tag. If you don't null the tag your branch will not get deployed. example:
```yaml
addons:
app1:
git:
tag: null
branch: "999-your-dev-branch-name"
```
1. After you have tested BigBang integration complete a Package MR and contact the codeowners to create a release tag. Package release tags follow the naming convention of {UpstreamChartVersion}-bb.{BigBangVersion} – example 1.2.3-bb.0.
1. Make sure to change the chart/values.yaml file to point to the new release tag rather than your dev branch (i.e. tag: "1.2.3-bb.0" in place of branch: "999-your-dev-branch-name"). example:
```yaml
addons:
app1:
git:
tag: "1.2.3-bb.0"
```
1. When you are done developing the BigBang chart features for your Package make a merge request in "Draft" status and add a label corresponding to your package name (must match the name in `values.yaml`). Also add any labels for dependencies of the package that are NOT core apps. The merge request will start a pipeline and use the labels to determine which addons to deploy. Fix any errors that appear in the pipeline. When the pipeline has pass and the MR is ready take it out of "Draft" and add the `status::review` label. Address any issues raised in the merge request comments.
## BigBang Development and Testing Cycle
There are two ways to test BigBang, imperative or GitOps with Flux. Your initial development can start with imperative testing. But you should finish with GitOps to make sure that your code works with Flux.
### Imperative
You can manually deploy bigbang with helm command line. With this method you can test local code changes without committing to a repository. Here are the steps that you can iterate with "code a little, test a little". You should have previously created the ../overrides directory as described in step #10 above. From the root of your local bigbang repo:
```shell
# Deploy with helm while pointing to your override values files.
# In this example the files are placed on your workstation at ../overrides/*
# Bigbang packages should create any needed secrets from the chart values
# If you have the values file encrypted with sops, temporarily decrypt it
helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../overrides/override-values.yaml -f ../overrides/registry-values.yaml -f ./chart/ingress-certs.yaml
# Conduct testing
# If you make code changes you can run another helm upgrade to pick up the new changes
helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../overrides/override-values.yaml -f ../overrides/registry-values.yaml -f ./chart/ingress-certs.yaml
# Tear down
helm delete bigbang -n bigbang
# Helm delete will not delete the bigbang namespace
kubectl delete ns bigbang
# Istio namespace will be stuck in "finalizing". So run the script to delete it.
hack/remove-ns-finalizer.sh istio-system
```
### GitOps with Flux
Using GitOps for development is NOT recommended. Your development iteration cycle time will be slowed down waiting for flux reconciliation. This is not an efficient use of your time. These instructions are included here for informational purposes. You can deploy your development code the same way a customer would deploy using GitOps. You must commit any code changes to your development branches because this is how GitOps works. There is a [customer template repository](https://repo1.dso.mil/platform-one/big-bang/customers/template) that has an example template for how to deploy using BigBang. You must fork or copy this repo to your own private repo. Make the necessary modifications as explained in the README.md. The setup information is not repeated here. Before committing code it is a good idea to manually run `helm template` and a `helm install` with dry run. This will reveal many errors before you make a commit. Here are the steps you can iterate:
```shell
# Verify chart code before committing
helm template bigbang ./chart -n bigbang -f ../customers/template/dev/configmap.yaml --debug
helm install bigbang ./chart -n bigbang -f ../customers/template/dev/configmap.yaml --dry-run
# Commit and push your code
# Deploy your bigbang template
kubectl apply -f dev/bigbang.yaml
# Monitor rollout
watch kubectl get pod,helmrelease -A
# Conduct testing
# Tear down
kubectl delete -f dev/bigbang.yaml
# Istio namespace will be stuck in "finalizing". So run the script to delete it. You will need 'jq' installed
hack/remove-ns-finalizer.sh istio-system
# If you have pushed code changes before the tear down, occasionally the bigbang deployments are not terminated because Flux has not had enough time to reconcile the helmreleases
# Re-deploy bigbang
kubectl apply -f dev/bigbang.yaml
# Run the sync script.
hack/sync.sh
# Tear down
kubectl delete -f dev/bigbang.yaml
hack/remove-ns-finalizer.sh istio-system
```
# Big Bang Integration: Overview
The following documents should be followed, in order, to fully integrate a new package into Big Bang:
1. [Upstream Helm Chart](./package-integration/package-integration-upstream.md): Initialize package workspace using an upstream Helm chart
1. [CICD Pipeline](./package-integration/package-integration-pipeline.md): Establish a baseline package pipeline for testing changes
1. [Flux Helm Chart](./package-integration/package-integration-flux.md): Create Flux compatible GitOps Helm chart required by Big Bang
1. [Service mesh](./package-integration/package-integration-service-mesh.md): Integrate with service mesh for ingress/egress
1. [Monitoring](./package-integration/package-integration-monitoring.md): Enable metrics scraping on product
1. [Database](./package-integration/package-integration-database.md): If required, add internal and external database support using Big Bang values
1. [Object Storage](./package-integration/package-integration-storage.md): If required, add internal or external object storage support using Big Bang values
1. [Single-sign On](./package-integration/package-integration-sso.md): If available, add single-sign on (SSO) through internal or external identify provider.
1. [Additional Tests](./package-integration/package-integration-testing.md): Add testing to validate basic functionality
1. [Network Policies](./package-integration/package-integration-network-policies.md): Add ingress/egress policies to restrict network traffic for security
1. [Policy Enforcement](./package-integration/package-integration-policy-enforcement.md): Update package to comply with default security and governance policies in Big Bang
1. [Supported Package](./package-integration/package-integration-supported.md): Migrate package into the Big Bang repo as a supported package
1. [Final Documentation](./package-integration/package-integration-documentation.md): Add additional Big Bang documentation for final release
docs/developer/package-integration/package-integration-database.md 0 → 100644
View file @ 256d554d
# Big Bang Package: Database Integration
If the package you are integrating connects to a database or cache server, you will need to follow the instructions below to integrate this feature into Big Bang
## Prerequisites
TBD
## Integration
## Validation
docs/developer/package-integration/package-integration-documentation.md 0 → 100644
View file @ 256d554d
# Big Bang Package: Documentation
Big Bang requires some additional documentation for supported packages to help user's understand how it interacts with other components. The following are documents that should be created or updated for integration into Big Bang:
- TBD
docs/developer/package-integration/package-integration-flux.md 0 → 100644
View file @ 256d554d
# Big Bang Package: Flux Integration
Big Bang uses a continuous deployment tool, [Flux](https://fluxcd.io/) to deploy packages using Helm charts sourced from Git ([GitOps](https://www.weave.works/technologies/gitops/)). This document will cover how to integrate a Helm chart, from a mission application or other package, into the Flux pattern required by Big Bang. Once complete, you will be able to deploy your package with Big Bang.
## Prerequisites
TBD
## Integration
## Validation
docs/developer/package-integration/package-integration-monitoring.md 0 → 100644
View file @ 256d554d
# Big Bang Package: Monitoring
Monitoring packages requires a way to scrape metrics, provide those to data storage, and analyzing the results. Big Bang uses Prometheus and Grafana as the service for monitoring. Most packages offer built-in Prometheus metrics scraping or an add-on that will scrape the metrics. This document will show you how to integrate metrics scraping with Big Bang.
## Prerequisites
TBD
## Integration
## Validation
docs/developer/package-integration/package-integration-network-policies.md 0 → 100644
View file @ 256d554d
# Big Bang Package: Network Policies
To help harden the Big Bang, network policies are put in place to only allow ingress and egress from package namespaces to other needed services. A deny by default policy is put in place to deny all traffic that is not explicitly allowed. The following is how to implement the network policies per Big Bang standards.
## Prerequisites
TBD
## Integration
## Validation
docs/developer/package-integration/package-integration-pipeline.md 0 → 100644
View file @ 256d554d
# Big Bang Package: Pipeline Integration
Big Bang contains a uses a continuous deployment tool to deploy packages using Helm charts sourced from Git. This document will cover how to integrate a Helm chart from a mission application or other package into the pattern Big Bang requires. Once complete, you will be able to deploy your package with Big Bang.
## Prerequisites
TBD
## Integration
## Validation
docs/developer/package-integration/package-integration-policy-enforcement.md 0 → 100644
View file @ 256d554d
# Big Bang Package: Policy Enforcement
Big Bang has several policies for Kubernetes resources to ensure best practices and security. For example, images must be pulled from Iron Bank, or containers must be run as non-root. These policies are currently enforced by [OPA Gatekeeper](https://repo1.dso.mil/platform-one/big-bang/apps/core/policy), which gets deployed as the first package in Big Bang.
When integrating your package, you must adhere to the policies that are enforced or your resources will be denied by the Kubernetes admission controller. The following is how to identify and fix policy violations.
## Prerequisites
TBD
## Integration
## Validation
docs/developer/package-integration/package-integration-service-mesh.md 0 → 100644
View file @ 256d554d
# Big Bang Package: Service Mesh Integration
[Istio](https://istio.io/) provides the [service mesh](https://istio.io/latest/about/service-mesh/) for Big Bang. The service mesh assists with secure traffic routing in the cluster. This document will show you how to update your package to support Big Bang's configuration of Istio.
## Prerequisites
TBD
## Integration
## Validation

UNCLASSIFIED - NO CUI