UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects
Commit a9a61025 authored by Michael McLeroy's avatar Michael McLeroy Committed by joshwolf
Browse files

Docs updates

parent 79656776
No related branches found
No related tags found
No related merge requests found
# CHANGELOG
# Big Bang Release Notes
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## 0.0.4
---
## [Unreleased]
### Changed
* Updated documentation
---
## [0.0.4] - 2020-12-16
### Changed
* Update Monitoring to [11.0.0-bb.2](https://repo1.dsop.io/platform-one/big-bang/apps/core/monitoring/-/tags/11.0.0-bb.2)
---
## [0.0.3] - 2020-12-15
## 0.0.3
### Added
* Documentation in [docs](./docs)
### Changed
* Updated Argo to [2.9.5-bb.1](https://repo1.dsop.io/platform-one/big-bang/apps/core/argocd/-/merge_requests/10) for Iron Bank images
* Updated Authservice to [0.1.3-bb.0](https://repo1.dsop.io/platform-one/big-bang/apps/sandbox/authservice/-/blob/master/CHANGELOG.md#013-bb0) for authservice secret generation: https://repo1.dsop.io/platform-one/big-bang/apps/sandbox/authservice/-/blob/master/CHANGELOG.md#013-bb0
* Updated ECK-Operator to [1.3.1-bb.1](https://repo1.dsop.io/platform-one/big-bang/apps/core/eck-operator/-/tags/1.3.0-bb.1)
* Updated Twistlock to [0.0.2-bb.0](https://repo1.dsop.io/platform-one/big-bang/apps/security-tools/twistlock/-/tags/0.0.2-bb.0) to add istio.enabled flag
* Updated Elasticsearch Kibana to [0.1.2-bb.0](https://repo1.dsop.io/platform-one/big-bang/apps/core/elasticsearch-kibana/-/tags/0.1.2-bb.0) and Pass istio.enabled to Elasticsearch Kibana
---
## [0.0.2] - 2020-12-11
### Added
* Initial release of Big Bang
---
\ No newline at end of file
......@@ -2,59 +2,79 @@
Thanks for taking the time to contribute to BigBang!
Table of Contents:
- [Contributing to Big Bang](#contributing-to-big-bang)
- [Developers Guide](#developers-guide)
- [Iron Bank Images](#iron-bank-images)
- [Local Kubernetes cluster](#local-kubernetes-cluster)
- [Deploying Big Bang (Quick Start)](#deploying-big-bang-quick-start)
- [Testing Big Bang Development Changes](#testing-big-bang-development-changes)
- [DNS](#dns)
- [Secrets & Certificates](#secrets--certificates)
- [Merge requests process](#merge-requests-process)
- [Pipeline Stages](#pipeline-stages)
- [Linting](#linting)
- [Smoke Testing](#smoke-testing)
- [Infrastructure Testing](#infrastructure-testing)
- [Network Creation](#network-creation)
- [Cluster(s) Creation](#clusters-creation)
- [Big Bang Installation](#big-bang-installation)
- [Big Bang Tests](#big-bang-tests)
- [Teardown](#teardown)
## Developers Guide
Big Bang is designed in such a way as to be as easily deployed locally as it is in production. In fact, most contributions begin locally.
Follow the steps below to get a complete local instantiation of Big Bang up locally using [k3d](https://k3d.io/).
## Iron Bank Images
Per the [charter](https://repo1.dsop.io/platform-one/big-bang/charter), all Big Bang packages will leverage container images from [IronBank](https://ironbank.dsop.io/). In order to pull these images, ImagePullSecrets must be provided to BigBang. To obtain access to these images, follow the guides below. These steps should NOT be used for production since the API keys for a user are only valid when the user is logged into [Registry1](https://registry1.dsop.io)
### Local Development Quickstart
1) Register for a free Ironbank account [Here](https://sso-info.il2.dsop.io/new_account.html)
1) Log into the [Iron Bank Registry](https://registry1.dsop.io), in the top right click your *Username* and then *User Profile* to get access to your *CLI secret*/API keys.
1) When installing BigBang, set the Helm Values `registryCredentials.username` and `registryCredentials.password` to match your Registry1 username and API token
#### Local `k3d` cluster
## Local Kubernetes cluster
Follow the steps below to get a local Kubernetes cluster for Big Bang using [k3d](https://k3d.io/).
```bash
# Create a local k3d cluster with the appropriate port forwards
k3d cluster create --k3s-server-arg "--disable=traefik" --k3s-server-arg "--disable=metrics-server" -p 80:80@loadbalancer -p 443:443@loadbalancer
```
#### Deploying Big Bang
The [Big Bang environment template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang/-/tree/master/bigbang) should be copied locally to start your deployment. Follow the instructions in the [template's readme](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang/-/tree/master/bigbang/README.md) and in the [Big Bang docs](./docs) for configuration.
## Deploying Big Bang (Quick Start)
```bash
# Deploy official, hardened fluxv2 from Iron Bank
# Alternatives:
# - Install non-hardened image: `flux install`
# - Install unofficial images from Big Bang repo: `flux install --registry registry.dsop.io/platform-one/big-bang/apps/sandbox/fluxv2`
hack/flux-install.sh
For development, it is quicker to test changes without having to push to Git. To do this, we can bypass Flux2 and deploy Big Bang directly with its Helm chart.
# Apply the development sops secret
# Modify sops-create.sh if you use your own SOPS secret
hack/sops-create.sh
Start by creating `myvalues.yaml` to configure your local Big Bang. Big Bang's template repository contains a starter [development values.yaml](https://repo1.dsop.io/platform-one/big-bang/customers/template/-/blob/main/dev/configmap.yaml).
# The above command creates the 'bigbang' namespace. If you skip it, create your own
kubectl create namespace bigbang
Configure `myvalues.yaml` to suit your needs.
# Apply the necessary dev secrets (e.g. pull secrets, certs)
# The .yaml files used for this are from the Big Bang environment template
sops -d bigbang/base/secrets.enc.yaml | kubectl apply -n bigbang -f -
sops -d bigbang/dev/secrets.enc.yaml | kubectl apply -n bigbang -f -
```bash
# Deploy the latest fluxv2 with Iron Bank images
# For development, you can use flux from the internet using 'flux install`
# Be aware, the internet version is likely newer than the Iron Bank version
./hack/flux-install.sh
# Apply a local version of the umbrella chart
# NOTE: This is the alternative to deploying a HelmRelease and having flux manage it, we use a local copy to avoid having to commit every change
# NOTE: Use yq to parse the kustomize values patch and pipe it to the helm values
# The .yaml files used for yq are from the Big Bang environment template
# NOTE: Flux will take care of the reconcilitation and retry loops for us, it is normal to see resources fail to deploy a few times on boot
yq m bigbang/prod/configmap.yaml bigbang/base/configmap.yaml | helm helm upgrade -i bigbang chart -n bigbang --create-namespace -f -
helm upgrade -i bigbang chart -n bigbang --create-namespace -f myvalues.yaml
# After making changes to the umbrella chart or values, you can update the chart idempotently
yq m bigbang/prod/configmap.yaml bigbang/base/configmap.yaml | helm helm upgrade -i bigbang chart -n bigbang --create-namespace -f -
# A convenience development script is provided to force fluxv2 to reconcile all helmreleases within the cluster
# A convenience development script is provided to force fluxv2 to reconcile all helmreleases within the cluster insteading of waiting for the next polling interval.
hack/sync.sh
```
#### DNS
For more extensive development, use the [Development Guide](docs/c_development.md).
## Testing Big Bang Development Changes
Development changes should be tested using a full GitOps environment. The [Big Bang environment template](https://repo1.dsop.io/platform-one/big-bang/customers/template/) should be replicated, either on a branch or new repository, to start your deployment. Follow the instructions in the [template's readme](https://repo1.dsop.io/platform-one/big-bang/customers/template/-/tree/main/README.md) and in the [Big Bang docs](./docs) for configuration.
Follow the [Big Bang documentation](./docs) for testing a full deployment of Big Bang.
## DNS
To ease with local development, the TLD `bigbang.dev` has been purchased with the following CNAME record:
......@@ -62,22 +82,13 @@ To ease with local development, the TLD `bigbang.dev` has been purchased with th
All routable endpoints BigBang deploys will use the TLD of `bigbang.dev` by default. It is expected that consumers modify this appropriately for their environment.
#### Secrets & Certificates
## Secrets & Certificates
A __development only__ gpg key is provided at `hack/bigbang-dev.asc` that is used to encrypt and decrypt the "secret" information in `hack/secrets`.
A __development only__ gpg key is provided at `bigbang-dev.asc` that is used to encrypt and decrypt the secrets in this Git repository (e.g. [hack/secrets](hack/secrets/).
We cannot stress enough, __do not use this key to encrypt real secret data__. It is a shared key meant to demonstrate the workflow of secrets management within Big Bang.
```bash
# Import the gpg key
gpg --import bigbang-dev.asc
# Decrypt the Big Bang Development Wildcard Cert
sops -d hack/secrets/ingress-cert.yaml
# Encrypt the Big Bang Development Wildcard Cert
sops -e hack/ingress-cert.yaml
```
Follow instructions in the [Big Bang encryption guide](docs/3_encryption.md) for how to encrypt and decrypt secrets.
## Merge requests process
......@@ -95,13 +106,13 @@ This stage is ran on every commit, and is a requirement for merging.
#### Smoke Testing
For fast feedback testing, an ephemeral in cluster pipeline is created using [k3d](https://k3d.io/) that lives for the lifetime of the gitlab ci job. Within that cluster, BigBang is deployed, and an initial set of smoke tests are performed against the deployment to ensure basic conformance.
For fast feedback testing, an ephemeral in cluster pipeline is created using [`k3d`](https://k3d.io) that lives for the lifetime of the gitlab ci job. Within that cluster, BigBang is deployed, and an initial set of smoke tests are performed against the deployment to ensure basic conformance.
This stage verifies several easy to check assumptions such as:
* does BigBang successfully install
* does BigBang successfully upgrade (from master)
* are endpoints routable
- does BigBang successfully install
- does BigBang successfully upgrade (from master)
- are endpoints routable
This stage also serves as a guide for local development, and care is taken to ensure all pipeline actions within this stage are repeatable locally.
......
# Umbrella
# Big Bang
Work in progress umbrella package
## Iron Bank Images
Per the Charter, all Big Bang packages will leverage container images from [IronBank](https://ironbank.dsop.io/). In order to pull these images, ImagePullSecrets must be provided to BigBang. For developers to obtain access
to the images, follow the guides below. These steps should NOT be used for production since the API keys for a user are only valid when the user is logged into [Registry1](https://registry1.dsop.io)
1) Register for a free Ironbank account [Here](https://sso-info.il2.dsop.io/new_account.html)
2) Log into the [Iron Bank Registry](https://registry1.dsop.io), in the top right click your *Username* and then *User Profile* to get access to your *CLI secret*/API keys.
3) When installing BigBang, set the Helm Values `registryCredentials.username` and `registryCredentials.password` to match your Registry1 username and API token
Big Bang is a declarative, continuous delivery tool for core DoD hardened and approved [packages](#packages) into a Kubernetes cluster. Big Bang follows a [GitOps](#gitops) approach to configuration management, using [Flux v2](#flux-v2) to reconcile Git with the cluster. Environments (e.g. dev, prod) and packages (e.g. istio) can be fully configured to suit the deployment needs.
## Usage
The following examples expect a cluster with fluxv2 preinstalled. This can be done by [installing the flux cli](https://toolkit.fluxcd.io/get-started/#install-the-flux-cli) and running `flux install`. This will install flux from the internet. If you wish to install the Iron Bank approved images, run `hack/flux-install.sh` to install flux from the [Iron Bank Registry](https://registry1.dsop.io).
Big Bang is intended to be used for deploying and maintaining a DoD hardened and approved set of packages into a Kubernetes cluster. Deployment and configuration of ingress/egress, load balancing, policy auditing, logging, monitoring, etc. are handled via Big Bang. Additional packages (e.g. ArgoCD, GitLab) can also be enabled and customized to extend Big Bang's baseline. Once deployed, the customer can use the Kubernetes cluster to add mission specific applications.
Additional information can be found in [Big Bang Overview](./docs/1_overview.md).
### Quickstart
## Getting Started
A quickstart BigBang environment template is provided [here](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang/-/tree/master/bigbang). See the README.md to get started.
To start using Big Bang, you will need to create your own Big Bang environment tailored to your needs. The [Big Bang customer template](https://repo1.dsop.io/platform-one/big-bang/customers/template/) is provided for you to copy into your own Git repository and begin modifications. Follow the instructions in [Big Bang Getting Started](./docs/2_getting_started.md) to customize and deploy Big Bang.
### Contributing
Please see our [contributing guide](./CONTRIBUTING.md) if you are interested in contributing to Big Bang.
\ No newline at end of file
Please see the [contributing guide](./CONTRIBUTING.md) if you are interested in contributing to Big Bang.
# Big Bang Release Notes
Table of Contents
- [Big Bang Release Notes](#big-bang-release-notes)
- [Version 0.0.2](#version-002)
- [What's New](#whats-new)
- [Bug Fixes](#bug-fixes)
---
## Version 0.0.2
Date: December 8th, 2020
### What's New
- Initial release of Big Bang
### Bug Fixes
- None
---
......@@ -5,7 +5,7 @@ Table of Contents
- [Big Bang Overview](#big-bang-overview)
- [Key Concepts](#key-concepts)
- [Big Bang](#big-bang)
- [Big Bang Umbrella](#big-bang-umbrella)
- [Big Bang Components](#big-bang-components)
- [Flux v2](#flux-v2)
- [GitOps](#gitops)
- [SOPS](#sops)
......@@ -25,9 +25,9 @@ Table of Contents
Big Bang is a declarative, continuous delivery tool for core DoD hardened and approved [packages](#packages) into a Kubernetes cluster. Big Bang follows a [GitOps](#gitops) approach to configuration management, using [Flux v2](#flux-v2) to reconcile Git with the cluster. Environments (e.g. dev, prod) and packages (e.g. istio) can be fully configured to suit the deployment needs.
### Big Bang Umbrella
### Big Bang Components
The umbrella is a sub-component of Big Bang that controls the deployment of all the packages. This is deployed via a Helm chart.
Big Bang is made of several components that operate together. These include a [base Kustomize](../base/), a [Helm chart](../chart), [packages](../chart/templates), and an [environment](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang). The environment and base Kustomize are used together to deploy the Big Bang configuration and Helm chart. The Helm chart is then used to facilitate deployment of the packages.
### Flux v2
......@@ -92,11 +92,11 @@ The diagram below shows a typcial deployment of Big Bang into a Kubernetes clust
1. Next, Kustomize is run on the environment configuration
1. The Kustomize files use Big Bang's Git repo as a base before applying overlays and patches for the configuration.
1. Flux uses SOPS to decrypt any secrets before deploying the manifests
1. After completing the Kustomization process, Flux deploys two ConfigMaps, two Secrets, and flux resources for Big Bang Umbrella
1. Big Bang's flux resources include a Git repository holding the umbrella Helm chart and a Helm Release resource that tells Flux how to deploy the Helm chart.
1. After completing the Kustomization process, Flux deploys two ConfigMaps, two Secrets, and flux resources for Big Bang
1. Big Bang's flux resources include a Git repository holding the Helm chart and a Helm Release resource that tells Flux how to deploy the Helm chart.
1. The repository is reconciled first, pulling the Helm chart from Git.
1. The Helm Release will check for the Helm chart and the Secrets / ConfigMaps deployed before performing a Helm install
1. Once the Helm release deploys the Helm chart for Big Bang umbrella, each package that is enabled will have a Flux Git Repository and Helm Release resource deployed.
1. Once the Helm release deploys the Helm chart for Big Bang, each package that is enabled will have a Flux Git Repository and Helm Release resource deployed.
1. All of the package Git repos containing Helm charts will be pulled so that Flux can reconcile dependencies.
1. Each package's Helm Release has dependencies built in. Flux will reconcile these dependencies and deploy the Helm chart for the package once all of the dependencies are ready.
1. Once all of the packages are ready, Big Bang will monitor Git periodically for changes and reconcile using the methods above.
......@@ -110,31 +110,31 @@ graph TD
style CustomVals fill:#00758f
style EncryptSecrets fill:#00758f
style PushToGit fill:#00758f
style DeployBB fill:#00758f
style DeployMan fill:#00758f
SetupSOPS(Setup SOPS keys) --> EncryptSecrets(Encrypt secrets)
SetupSOPS --> CustomVals(Customize values)
CustomSecrets(Customize secrets) --> EncryptSecrets
CustomVals --> PushToGit(Push customization to Git)
EncryptSecrets --> PushToGit
PushToGit --> DeployBB(Deploy BigBang Manifest)
PushToGit --> DeployMan(Deploy BigBang Manifest)
DeployBB --> KustResEnv[[Deploy Environment Kustomization Resource]]
KustResEnv --> HelmResUmb[[Deploy Umbrella Helm Release Resource]]
DeployBB --> GitResEnv[[Deploy Environment Git Repository Resource]]
KustResEnv --> GitResUmb[[Deploy Umbrella Git Repository Resource]]
DeployMan --> KustResEnv[[Deploy Environment Kustomization Resource]]
KustResEnv --> HelmResBB[[Deploy Big Bang Helm Release Resource]]
DeployMan --> GitResEnv[[Deploy Environment Git Repository Resource]]
KustResEnv --> GitResBB[[Deploy Big Bang Git Repository Resource]]
KustResEnv --> SOPS
GitResEnv --> PullEnv[[Pull environment]]
PullEnv --> SOPS[[SOPS Decrypt secrets]]
SOPS --> DeployVals[[Deploy ConfigMap and Secrets]]
GitResUmb --> PullUmb[[Pull Umbrella Helm Chart]]
PullUmb --> DeployUmb[[Deploy Umbrella w/ Helm Chart]]
HelmResUmb --> DeployUmb
DeployVals --> DeployUmb
GitResBB --> PullBB[[Pull Big Bang Helm Chart]]
PullBB --> DeployBB[[Deploy Big Bang Helm Chart]]
HelmResBB --> DeployBB
DeployVals --> DeployBB
DeployUmb --> PackGit[[Deploy Package Git Repository Resources]]
DeployBB --> PackGit[[Deploy Package Git Repository Resources]]
PackGit --> PackPull[[Pull Package Helm Charts]]
PackPull --> PackDep[[Resolve Package Dependencies]]
PackDep --> PackReady{Package Ready?}
......
......@@ -56,7 +56,7 @@ Table of Contents
## Configuration Template
A [template for configuring multi-cluster environments](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang/-/tree/master/bigbang) is provided to assist with getting Big Bang setup correctly. You should make a copy of the `./bigbang` folder from the [customer Big Bang repository](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang) and place it into a Git repository under your control.
A [template for configuring multi-cluster environments](https://repo1.dsop.io/platform-one/big-bang/customers/template/) is provided to assist with getting Big Bang setup correctly. You should make a copy of the `./bigbang` folder from the [customer Big Bang repository](https://repo1.dsop.io/platform-one/big-bang/customers/template) and place it into a Git repository under your control.
### Overview
......@@ -68,7 +68,7 @@ The template is setup to allow you to customize the Big Bang deployment for your
- Reference to SOPS private key - See the [encryption help](3_encryption.md) for more information.
- Iron Bank pull credentials
The [Configuration Template help](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang/-/blob/master/bigbang/README.md) contains details on how to setup these items.
The [Configuration Template help](https://repo1.dsop.io/platform-one/big-bang/customers/template/-/blob/main/README.md) contains details on how to setup these items.
If there is additional configuration you want, refer to the [configuration help](4_configuration.md) for details.
......
......@@ -107,7 +107,7 @@ TBD - [This article](https://blog.doit-intl.com/injecting-secrets-from-aws-gcp-o
## Configure Big Bang
Big Bang needs to know how to retrieve the private key so it can deploy the encrypted secrets from Git. Decryption configuration is placed in the top-level manifest (e.g. `dev.yaml`, `prod.yaml`) from the [Big Bang template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang). By default, the `Kustomization` resource uses a Secret named `sops-gpg` for the private key as shown here:
Big Bang needs to know how to retrieve the private key so it can deploy the encrypted secrets from Git. Decryption configuration is placed in the top-level manifest (e.g. `dev.yaml`, `prod.yaml`) from the [Big Bang template](https://repo1.dsop.io/platform-one/big-bang/customers/template). By default, the `Kustomization` resource uses a Secret named `sops-gpg` for the private key as shown here:
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
......
......@@ -19,7 +19,7 @@ Table of Contents
## Overview
Configuration of Big Bang is achieved by overriding default values set in the package or Big Bang using the [environment template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang). The template has a 4 potential locations for setting values: `base/secrets.enc.yaml`, `base/configmap.yaml`, `<env>/secrets.enc.yaml`, and `<env>/configmap.yaml`. Overrides proceed as follows, with `<env>/configmap.yaml` having the highest precedence.
Configuration of Big Bang is achieved by overriding default values set in the package or Big Bang using the [environment template](https://repo1.dsop.io/platform-one/big-bang/customers/template). The template has a 4 potential locations for setting values: `base/secrets.enc.yaml`, `base/configmap.yaml`, `<env>/secrets.enc.yaml`, and `<env>/configmap.yaml`. Overrides proceed as follows, with `<env>/configmap.yaml` having the highest precedence.
```mermaid
graph TD
......@@ -31,7 +31,7 @@ graph TD
-->env-c[`<env>/configmap.yaml` values]
```
In all four cases, Big Bang reads a single key named `values.yaml` that contains the data to override. See the [Big Bang environment template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang) for examples on how to use these files to override values.
In all four cases, Big Bang reads a single key named `values.yaml` that contains the data to override. See the [Big Bang environment template](https://repo1.dsop.io/platform-one/big-bang/customers/template) for examples on how to use these files to override values.
## Pre-configuration
......@@ -54,7 +54,7 @@ At a minimum, the following items must be configured for a default Big Bang depl
- [SOPS private key reference](3_encryption.md).
- [Registry pull credentials](#registry-pull-credentials)
The Big Bang [Environment Template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang) has placeholders for all of the above.
The Big Bang [Environment Template](https://repo1.dsop.io/platform-one/big-bang/customers/template) has placeholders for all of the above.
## Big Bang Globals
......@@ -111,10 +111,10 @@ Big Bang deploys four flux resources that can be customized:
|--|--|--|
| GitRepository | Environment | Top-level manifest (e.g. `dev.yaml`, `prod.yaml`)
| Kustomization | Environment | Top-level manifest (e.g. `dev.yaml`, `prod.yaml`)
| GitRepostiory | Big Bang Umbrella | [Link](../base/gitrepository.yaml) |
| HelmRelease | Big Bang Umbrealla | [Link](../base/helmrelease.yaml) |
| GitRepostiory | Big Bang | [Link](../base/gitrepository.yaml) |
| HelmRelease | Big Bang | [Link](../base/helmrelease.yaml) |
In addition, each package contains its own GitRepository and HelmRelease resource that can be customized. Look in the [Umbrella templates](../chart/templates) for the these resources.
In addition, each package contains its own GitRepository and HelmRelease resource that can be customized. Look in the [Helm chart templates](../chart/templates) for the these resources.
Settings for eny of these resources can be overridden by [patching](https://kubectl.docs.kubernetes.io/references/kustomize/patches/) the resource in your environment's kustomization files. Use Flux's documentation for [GitRepository](https://toolkit.fluxcd.io/components/source/gitrepositories/), [HelmRelease](https://toolkit.fluxcd.io/components/helm/helmreleases/), and [Kustomization](https://toolkit.fluxcd.io/components/kustomize/kustomization/) to find settings for these resources.
......@@ -130,7 +130,7 @@ patchesStrategicMerge:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: umbrella-repo
name: bigbang
spec:
ref:
$patch: replace
......@@ -151,9 +151,9 @@ metadata:
namespace: bigbang
spec:
interval: 1m
url: https://repo1.dsop.io/platform-one/big-bang/customers/bigbang.git
url: https://repo1.dsop.io/platform-one/big-bang/customers/template.git
ref:
branch: master
branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
......@@ -165,7 +165,7 @@ spec:
sourceRef:
kind: GitRepository
name: environment-repo
path: ./bigbang/dev
path: ./dev
prune: true
decryption:
provider: sops
......@@ -177,13 +177,13 @@ spec:
If you have pull credentials for your docker registry, add them to `secrets.enc.yaml`. Here is an example:
> The name of the Secret must be `secrets` or `secrets-common` for Big Bang to read values from it.
> The name of the Secret must be `common-bb` or `environment-bb` for Big Bang to read values from it.
```yaml
apiVersion: v1
kind: Secret
metadata:
name: secrets-common
name: common-bb
stringData:
values.yaml: |-
registryCredentials:
......@@ -216,12 +216,12 @@ gatekeeper:
You will also need to merge this file with the existing configmaps in `kustomization.yaml`.
> The name of the ConfigMap must be `configmap` or `configmap-common` for Big Bang to read values from it.
> The name of the ConfigMap must be `common` or `environment` for Big Bang to read values from it.
```yaml
namespace: bigbang
configMapGenerator:
- name: configmap
- name: common
behavior: merge
files:
- values.yaml=configmap.yaml
......
......@@ -44,7 +44,7 @@ Big Bang follows a [GitOps](https://www.weave.works/blog/what-is-gitops-really)
## Upgrade
All changes to the Big Bang cluster should be made through Git. After changes are pushed, Big Bang will automatically reconcile the difference with the cluster.
All changes to the Big Bang cluster should be made through Git. After changes are pushed, Big Bang will automatically reconcile the difference with the cluster.
> It may take Big Bang up to 10 minutes to recognize your changes and start to deploy them. This is based on the `interval` value set for polling. You can force Big Bang to immediately check for changes by running the [sync.sh](../hack/sync.sh) script.
......@@ -74,7 +74,7 @@ The following commands will help you monitor the progress of the Big Bang deploy
# `environment-repo`: STATUS should be True
NAMESPACE NAME URL READY STATUS AGE
bigbang environment-repo https://repo1.dsop.io/platform-one/big-bang/customers/bigbang.git True Fetched revision: master/185e252f4452d897531ab0314adc7a189562be31 2m7s
bigbang environment-repo https://repo1.dsop.io/platform-one/big-bang/customers/template.git True Fetched revision: main/185e252f4452d897531ab0314adc7a189562be31 2m7s
```
1. Verify the environment Kustomization properly worked
......@@ -84,7 +84,7 @@ The following commands will help you monitor the progress of the Big Bang deploy
# `environment`: READY should be True
NAMESPACE NAME READY STATUS AGE
bigbang environment True Applied revision: master/185e252f4452d897531ab0314adc7a189562be31 6m41s
bigbang environment True Applied revision: main/185e252f4452d897531ab0314adc7a189562be31 6m41s
```
1. Verify the ConfigMaps were deployed
......@@ -92,10 +92,10 @@ The following commands will help you monitor the progress of the Big Bang deploy
```bash
kubectl get configmap -l kustomize.toolkit.fluxcd.io/namespace -A
# 'configmap' and 'configmap-common' should exist
# 'common' and 'environment' should exist
NAMESPACE NAME DATA AGE
bigbang configmap-cch6942dk9 1 19m
bigbang configmap-common-d2tgb27f56 1 19m
bigbang common-cch6942dk9 1 19m
bigbang environment-d2tgb27f56 1 19m
```
1. Verify the Secrets were deployed
......@@ -103,10 +103,10 @@ The following commands will help you monitor the progress of the Big Bang deploy
```bash
kubectl get secrets -l kustomize.toolkit.fluxcd.io/namespace -A
# 'secrets' and 'secrets-common' should exist
# 'common-bb' and 'environment-bb' should exist
NAMESPACE NAME TYPE DATA AGE
bigbang secrets-common-kc5t8dbdfh Opaque 1 18m
bigbang secrets-mhddkt46bd Opaque 1 18m
bigbang common-bb-kc5t8dbdfh Opaque 1 18m
bigbang environment-bb-mhddkt46bd Opaque 1 18m
```
1. Verify the Big Bang Helm Chart was pulled
......@@ -114,9 +114,9 @@ The following commands will help you monitor the progress of the Big Bang deploy
```bash
kubectl get gitrepositories -A
# 'umbrella-repo' READY should be True
# 'bigbang' READY should be True
NAME URL READY STATUS AGE
umbrella-repo https://repo1.dsop.io/platform-one/big-bang/umbrella.git True Fetched revision: master/8a4a1ddd0c9edf316f5362680cf2921baf0c3451 25m
bigbang https://repo1.dsop.io/platform-one/big-bang/umbrella.git True Fetched revision: master/8a4a1ddd0c9edf316f5362680cf2921baf0c3451 25m
```
1. Verify the Big Bang Helm Chart was deployed
......@@ -124,9 +124,9 @@ The following commands will help you monitor the progress of the Big Bang deploy
```bash
kubectl get hr -A
# 'umbrella-helm' READY should be True
# 'bigbang' READY should be True
NAMESPACE NAME READY STATUS AGE
bigbang umbrella-helm True Release reconciliation succeeded 28m
bigbang bigbang True Release reconciliation succeeded 28m
```
1. Verify Big Bang package Helm charts are pulled
......@@ -137,7 +137,7 @@ The following commands will help you monitor the progress of the Big Bang deploy
# The Git repository holding the Helm charts for each package can be seen in the URL column.
# The STATUS column shows the branch and tag of the revision being used.
NAMESPACE NAME URL READY STATUS AGE
bigbang umbrella-chart https://repo1.dsop.io/platform-one/big-bang/apps/sandbox/umbrella True Fetched revision: master/3a44686520152e576a8c2c6f264876efff497c4b 8m25s
bigbang bigbang https://repo1.dsop.io/platform-one/big-bang/umbrella.git True Fetched revision: master/3a44686520152e576a8c2c6f264876efff497c4b 8m25s
bigbang logging https://repo1.dsop.io/platform-one/big-bang/apps/core/logging.git True Fetched revision: release-v0.2.x/9cfe1e14c12098464ee89eb877614f781cd78fb7 8m23s
bigbang certmanager https://repo1.dsop.io/platform-one/big-bang/apps/sandbox/cert-manager.git True Fetched revision: release-v1.0.x/1247135baf145dcfad4a4a02ef679c48fb76d9fb 8m23s
bigbang istio https://repo1.dsop.io/platform-one/big-bang/apps/core/servicemesh.git True Fetched revision: chart-release/2b02a51b7950ce21bac26403fa25d09e7e3f86c3 8m23s
......
......@@ -101,7 +101,7 @@ kubectl get events --field-selector involvedObject.kind=Kustomization -A
| `kustomize build failed: json: unknown field` | There is a syntax error with the kustomization files. | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. Fix the error and push to Git. |
| `evalsymlink failure ... no such file or directory` | A reference to a file in `kustomization.yaml` is incorrect | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. Fix the error and push to Git. |
| `Error: accumulating resources ...` | A reference to a base is incorrect | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem.Review the `bases:` section for correct paths to find the error. Fix the error and push to Git. |
| `Error fetchingref: fatal: couldn't find remote ref ...` | The branch, tag, or sha used for a remote base is incorrect | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. It is likely the remote reference to the Big Bang umbrella in the `base` folder. Review the `bases:` section for correct paths to find the error. Fix the error and push to Git. |
| `Error fetchingref: fatal: couldn't find remote ref ...` | The branch, tag, or sha used for a remote base is incorrect | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. It is likely the remote reference to the Big Bang's Kustomize in the `base` folder. Review the `bases:` section for correct paths to find the error. Fix the error and push to Git. |
| `Error: merging from generator ...` | Kustomize is trying to merge with a resource that is non-existant. This is usually due to naming the merging `ConfigMap` or `Secret` incorrectly compared to a base `ConfigMap` or `Secret`. | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. Look for the keyword `merge` in the `kustomization.yaml` files and verify the `name` is correctly set. |
## Packages
......
# Appendix C - Big Bang Development
## So you want to develop on Big Bang Umbrella?
## So you want to develop on Big Bang?
Included here is a setup that will allow you to checkout and begin development using your workstation and a minimal EC2 instance in AWS.
......@@ -8,7 +8,7 @@ Included here is a setup that will allow you to checkout and begin development u
+ AWS access (with permissions to create an EC2 instance)
+ Flux CLI installed on your local machine
+ Access to the Umbrella Git Repo
+ Access to the Git Repo
+ kubectl installed on local machine
+ yq installed on local machine
......@@ -128,33 +128,100 @@ optional:
cat ~/.kube/config
```
+ Move to your workstation and install Big Bang Umbrella on the cluster
+ Move to your workstation and setup namespace
```bash
# Test to see if you can connect to your cluster
kubectl get nodes
# From the base of the project
flux install
kubectl create ns bigbang
```
kubectl apply -f tests/ci/shared-secrets.yaml
+ Customize your Helm values
# Helm install BigBang umbrella
```bash
# You will be overriding values in `chart/values.yaml` for development
# You can use the [Big Bang template's dev ConfigMap](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang/-/blob/template/bigbang/dev/configmap.yaml) to start. This will minimize the resources for deploying BigBang.
# For convenience, it is also copied here
cat << EOF > my-values.yaml
hostname: bigbang.dev
flux:
interval: 1m
rollback:
cleanupOnFail: false
gatekeeper:
values:
replicas: 1
istio:
values:
kiali:
dashboard:
auth:
strategy: anonymous
ingressGateway:
serviceAnnotations:
# Ensure mission apps have internal load balancer only
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
# Enable cross zone load balancing
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
logging:
values:
elasticsearch:
master:
count: 1
persistence:
size: 5Gi
resources:
limits:
cpu: 1
memory: 3Gi
data:
count: 1
persistence:
size: 5Gi
resources:
limits:
cpu: 1
memory: 3Gi
twistlock:
values:
console:
persistence:
size: 5Gi
EOF
# Method 1 - go for it. (Note: You don't need to set registryCredentials if you configured registry pull secret on the cluster in previous steps)
# Add any additional development values to this file as needed
# You can add registry1 pull credentials here for development
# Examples included enabling add-ons, disabling unneeded features, etc.
```
yq r examples/complete/envs/dev/patch-bigbang.yaml 'spec.values' | helm upgrade -i bigbang chart -n bigbang --create-namespace --set registryCredentials.username='<your user>' --set registryCredentials.password=<your cli key> -f -
+ Deploy secrets
# Method 2 - Modify some values
```bash
# These are all OPTIONAL. Deploy them if you need them
yq r examples/complete/envs/dev/patch-bigbang.yaml 'spec.values' > my-values.yaml
# Deploy the bigbang-dev.asc SOPS key into the bigbang namespace
./hack/create-sops.sh
# Modify my-values.yaml
# Install using your new values. You could also modify the values in place. (Note: You don't need to set registryCredentials if you configured registry pull secret on the cluster in previous steps)
# Deploy the authservice configuration
sops -d ./hack/secrets/authservice-config.yaml | kubectl apply -f -
# Deploy the ingress certificates
sops -d ./hack/secrets/ingress-cert.yaml | kubectl apply -f -
# Apply tests CI shared secrets
kubectl apply -f tests/ci/shared-secrets.yaml
```
+ Install BigBang
```bash
# Helm install BigBang
helm upgrade -i bigbang chart -n bigbang --create-namespace --set registryCredentials.username='<your user>' --set registryCredentials.password=<your cli key> -f my-values.yaml
```
......@@ -162,8 +229,8 @@ helm upgrade -i bigbang chart -n bigbang --create-namespace --set registryCreden
+ You can now modify your local /etc/hosts files (Or whatever the Windows people call it these days)
```bash
160.1.38.137 kibana.bigbang.dev
160.1.38.137 kiali.bigbang.dev
160.1.38.137 kibana.bigbang.dev
160.1.38.137 kiali.bigbang.dev
160.1.38.137 prometheus.bigbang.dev
160.1.38.137 graphana.bigbang.dev
```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment