UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects
Commit a54f7689 authored by Micah Nagel's avatar Micah Nagel
Browse files

Merge branch '1242-improve-docs-flow' into 'master'

Fix broken links

Closes #1242

See merge request platform-one/big-bang/bigbang!1872
parents 6b79b223 f86a9d43
No related branches found
No related tags found
1 merge request!1872Fix broken links
Pipeline #886649 passed with warnings
...@@ -6,12 +6,6 @@ Table of Contents: ...@@ -6,12 +6,6 @@ Table of Contents:
- [Contributing to Big Bang](#contributing-to-big-bang) - [Contributing to Big Bang](#contributing-to-big-bang)
- [Developers Guide](#developers-guide) - [Developers Guide](#developers-guide)
- [Local Git Setup](#local-git-setup)
- [Pre-commit hooks](#pre-commit-hooks)
- [Local Setup](#local-setup)
- [Prereqs](#prereqs)
- [Steps](#steps)
- [Combining Multiple Commits](#combining-multiple-commits)
- [Iron Bank Images](#iron-bank-images) - [Iron Bank Images](#iron-bank-images)
- [Local Kubernetes cluster](#local-kubernetes-cluster) - [Local Kubernetes cluster](#local-kubernetes-cluster)
- [Deploying Big Bang (Quick Start)](#deploying-big-bang-quick-start) - [Deploying Big Bang (Quick Start)](#deploying-big-bang-quick-start)
......
...@@ -218,7 +218,7 @@ Repository: [Cluster Auditor](https://repo1.dso.mil/platform-one/big-bang/apps/c ...@@ -218,7 +218,7 @@ Repository: [Cluster Auditor](https://repo1.dso.mil/platform-one/big-bang/apps/c
Dependencies: Dependencies:
* [Logging](#Logging) * [Logging](#logging)
* [OPA Gatekeer](#policy-enforcement) * [OPA Gatekeer](#policy-enforcement)
Owners: Owners:
......
...@@ -23,4 +23,5 @@ ...@@ -23,4 +23,5 @@
- Our goal is to have these docs available on a webpage hosted on the BigBang Cluster using Hugo - Our goal is to have these docs available on a webpage hosted on the BigBang Cluster using Hugo
(https://docs.bigbang.dev by default) (look [here](./PackageDocumentation.md) for more info) (https://docs.bigbang.dev by default) (look [here](./PackageDocumentation.md) for more info)
- Currently the docs are hosted for consumption on https://docs-bigbang.dso.mil/
- This allows us to support a centralized location for package configuration documentation, while allowing support for dynamically added versions of distributed packages in a maintainable way. - This allows us to support a centralized location for package configuration documentation, while allowing support for dynamically added versions of distributed packages in a maintainable way.
...@@ -16,8 +16,8 @@ The Flux2 Engine has native Helm support, meaning the controller deployed as par ...@@ -16,8 +16,8 @@ The Flux2 Engine has native Helm support, meaning the controller deployed as par
Argo, has taken the ownership of rendering and managing the lifecycle of applications that does not work exactly as expected by helm. As a result, there are several vendor Helm Charts that **do not deploy successfully** with Argo because of how Argo shims Helm Hooks to Argo specific sync phases. Argo, has taken the ownership of rendering and managing the lifecycle of applications that does not work exactly as expected by helm. As a result, there are several vendor Helm Charts that **do not deploy successfully** with Argo because of how Argo shims Helm Hooks to Argo specific sync phases.
* GitLab initial secret creation is performed via a [subchart]([https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/charts/shared-secrets](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/charts/shared-secrets)) * GitLab initial secret creation is performed via a [subchart](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/charts/shared-secrets)
* Kube Prometheus Stack - [prometheusrule admission webhook]([https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusrules-admission-webhooks](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusrules-admission-webhooks)) is created via a helm `install` hook * Kube Prometheus Stack - [prometheusrule admission webhook](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusrules-admission-webhooks) is created via a helm `install` hook
* Confluent - A deletion hook is part of a subchart gets run at different point in the lifecycle * Confluent - A deletion hook is part of a subchart gets run at different point in the lifecycle
As new features of Helm get developed and leveraged by the community, we would need to lean on the time and availability of the Argo developers to re-implement the capabilities. As new features of Helm get developed and leveraged by the community, we would need to lean on the time and availability of the Argo developers to re-implement the capabilities.
......
...@@ -90,7 +90,7 @@ addons: ...@@ -90,7 +90,7 @@ addons:
### Single Sign On ### Single Sign On
Anchore Enterprise 2.1+ can be configured to support user login to the UI using identities from external identity providers that support SAML 2.0. In such a configuration, Anchore never stores any credentials for the users, only their usernames and Anchore permissions, and all UI access is gated through a user’s valid login into the identity provider. Anchore uses the external provider to verify username identity and initialize a username, account, and roles on first login for a new user. Once a user’s identity is initialized in Anchore, the Anchore administrator may manage user permissions by managing the roles associated with the user’s identity in Anchore itself. For more information, see [Anchore Enterprise SSO Support](https://docs.anchore.com/current/docs/overview/sso/). Anchore Enterprise 2.1+ can be configured to support user login to the UI using identities from external identity providers that support SAML 2.0. In such a configuration, Anchore never stores any credentials for the users, only their usernames and Anchore permissions, and all UI access is gated through a user’s valid login into the identity provider. Anchore uses the external provider to verify username identity and initialize a username, account, and roles on first login for a new user. Once a user’s identity is initialized in Anchore, the Anchore administrator may manage user permissions by managing the roles associated with the user’s identity in Anchore itself. For more information, see [Anchore Enterprise SSO Support](https://docs.anchore.com/current/docs/configuration/sso/).
See below for an example of the values to provide to Anchore Enterprise for SSO setup: See below for an example of the values to provide to Anchore Enterprise for SSO setup:
...@@ -105,7 +105,7 @@ addons: ...@@ -105,7 +105,7 @@ addons:
### Storage ### Storage
Anchore relies on a PostgreSQL database as its primary data store. By default, Anchore will deploy an in-cluster PostgreSQL database, but it is recommended that an external PostgreSQL 9.6+ database be used, which can be configured in the Big Bang values.yaml. For more information, see [Anchore Enterprise Storage Overview](https://docs.anchore.com/current/docs/installation/storage/). Anchore relies on a PostgreSQL database as its primary data store. By default, Anchore will deploy an in-cluster PostgreSQL database, but it is recommended that an external PostgreSQL 9.6+ database be used, which can be configured in the Big Bang values.yaml. For more information, see [Anchore Enterprise Storage Overview](https://docs.anchore.com/current/docs/configuration/storage/).
### High Availability ### High Availability
...@@ -135,7 +135,7 @@ _Note:_ within Big Bang, logs are captured by fluentbit and shipped to elastic b ...@@ -135,7 +135,7 @@ _Note:_ within Big Bang, logs are captured by fluentbit and shipped to elastic b
### Monitoring ### Monitoring
Anchore Engine and Enterprise expose prometheus metrics in the API of each service if the config.yaml used by that service has the metrics.enabled key set to true. Each service exports its own metrics and is typically scraped by a Prometheus installation to gather the metrics. Anchore does not aggregate or distribute metrics between services. You should configure your Prometheus deployment or integration to check each Anchore service’s api (using the same port it exports), for the /metrics route. For more information, see [Anchore Enterprise Monitoring](https://docs.anchore.com/current/docs/monitoring/#monitoring-in-kubernetes-andor-helm-chart). Anchore Engine and Enterprise expose prometheus metrics in the API of each service if the config.yaml used by that service has the metrics.enabled key set to true. Each service exports its own metrics and is typically scraped by a Prometheus installation to gather the metrics. Anchore does not aggregate or distribute metrics between services. You should configure your Prometheus deployment or integration to check each Anchore service’s api (using the same port it exports), for the /metrics route. For more information, see [Anchore Enterprise Monitoring](https://docs.anchore.com/current/docs/configuration/monitoring/#monitoring-in-kubernetes-andor-helm-chart).
The Big Bang Anchore Helm chart has been modified to use your `monitoring:` values in Big Bang to automatically toggle metrics on/off. The Big Bang Anchore Helm chart has been modified to use your `monitoring:` values in Big Bang to automatically toggle metrics on/off.
......
...@@ -13,7 +13,7 @@ The MinIO tenants are created using the [MinIO package](https://repo1.dso.mil/pl ...@@ -13,7 +13,7 @@ The MinIO tenants are created using the [MinIO package](https://repo1.dso.mil/pl
The final package is the MinIO console. This is a graphical user interface that allows management of an individual tenant. The official package can be found [here](https://repo1.dso.mil/platform-one/big-bang/apps/application-utilities/minio). The final package is the MinIO console. This is a graphical user interface that allows management of an individual tenant. The official package can be found [here](https://repo1.dso.mil/platform-one/big-bang/apps/application-utilities/minio).
![Tenant Architecture](https://repo1.dso.mil/platform-one/big-bang/apps/application-utilities/minio-operator/-/raw/main/upstream/operator/docs/images/architecture.png) [Tenant Architecture](https://raw.githubusercontent.com/minio/operator/master/docs/images/architecture.png)
Note: The Minio Operator needs to be able to reach out to the minio instances. This is to ensure that on an upgrade all existing pools are shut down before starting new ones. If you run into issues with upgrades ensure that networkPolicies allow ingress to the minio pods in your namespace on port 9000. Note: The Minio Operator needs to be able to reach out to the minio instances. This is to ensure that on an upgrade all existing pools are shut down before starting new ones. If you run into issues with upgrades ensure that networkPolicies allow ingress to the minio pods in your namespace on port 9000.
......
...@@ -32,4 +32,4 @@ The following is a general overview of the process, the [deployment guides](guid ...@@ -32,4 +32,4 @@ The following is a general overview of the process, the [deployment guides](guid
## New User Orientation ## New User Orientation
* New users are encouraged to read through the Useful Background Contextual Information present in the [understanding_bigbang folder](./understanding_bigbang) * New users are encouraged to read through the Useful Background Contextual Information present in the [understanding-bigbang folder](./understanding-bigbang)
...@@ -252,7 +252,7 @@ If you need to handle mirroring manually, there is an example Ansible script pro ...@@ -252,7 +252,7 @@ If you need to handle mirroring manually, there is an example Ansible script pro
#### Konvoy Cluster #### Konvoy Cluster
Modify the `cluster.yaml` file and apply. More details can be found on the [D2iQ Konvoy documentation](https://docs.d2iq.com/dkp/konvoy/1.6/install/install-airgapped/). Modify the `cluster.yaml` file and apply. More details can be found on the [D2iQ Konvoy documentation](https://docs.d2iq.com/dkp/konvoy/1.8/install/install-airgapped/).
```yaml ```yaml
kind: ClusterConfiguration kind: ClusterConfiguration
......
...@@ -62,7 +62,8 @@ Follow the instructions from the script output to access and use the cluster. ...@@ -62,7 +62,8 @@ Follow the instructions from the script output to access and use the cluster.
## Install FluxCD ## Install FluxCD
The Big Bang product is tightly coupled with the GitOps tool FluxCD. Before you can deploy Big Bang you must deploy FluxCD on your k8s cluster. To guarantee that you are using the version of FluxCD that is compatible with the version of Big Bang that you are deploying use the Big Bang provided [script](./scripts/install_flux.sh). You will need your Iron Bank pull credentials and command line access to the k8s cluster from your workstation. The Big Bang product is tightly coupled with the GitOps tool FluxCD. Before you can deploy Big Bang you must deploy FluxCD on your k8s cluster. To guarantee that you are using the version of FluxCD that is compatible with the version of Big Bang that you are deploying use the Big Bang provided [script](../../scripts/install_flux.sh). You will need your Iron Bank pull credentials and command line access to the k8s cluster from your workstation.
```shell ```shell
./scripts/install_flux.sh -u your-user-name -p your-password ./scripts/install_flux.sh -u your-user-name -p your-password
``` ```
......
...@@ -4,9 +4,9 @@ ...@@ -4,9 +4,9 @@
BigBang developers use [k3d](https://k3d.io/), a lightweight wrapper to run [k3s](https://github.com/rancher/k3s) (Rancher Lab’s minimal Kubernetes distribution) in Docker. K3d is a virtualized kubernetes cluster that is quick to start and tear down for fast development iteration. K3d is sufficient for 95% of BigBang development work. In limited cases developers will use real infrastructure k8s deployments with Rancher, Konvoy, EKS, etc. Only k3d is covered in this document. BigBang developers use [k3d](https://k3d.io/), a lightweight wrapper to run [k3s](https://github.com/rancher/k3s) (Rancher Lab’s minimal Kubernetes distribution) in Docker. K3d is a virtualized kubernetes cluster that is quick to start and tear down for fast development iteration. K3d is sufficient for 95% of BigBang development work. In limited cases developers will use real infrastructure k8s deployments with Rancher, Konvoy, EKS, etc. Only k3d is covered in this document.
It is not recommend to run k3d with Big Bang on your local computer. Instead use a remote k3d cluster running on an EC2 instance to shift the compute and network bandwidth to the cloud. Big Bang can be quite resource intensive and it requires a huge download bandwidth for the images. If you do insist on running k3d locally you should disable certain packages before deploying. You can do this in the values.yaml file by setting the package deploy to false. One of the packages that is most resource-intensive is the logging package. And you should create a local image registry cache to minimize the amount of image downloading. It is not recommend to run k3d with Big Bang on your local workstation. Instead use a remote k3d cluster running on an EC2 instance to shift the compute and network bandwidth to the cloud. Big Bang can be quite resource intensive and it requires a huge download bandwidth for the images. If you do insist on running k3d locally you should disable certain packages before deploying. You can do this in the values.yaml file by setting the package deploy to false. One of the packages that is most resource-intensive is the logging package. And you should create a local image registry cache to minimize the amount of image downloading.
There is a script [/docs/developer/scripts/k3d-dev.sh](./scripts/) that automates the creation and teardown of a remote k3d development environment. Run the script from your workstation after [installing dependencies](/docs/developer/scripts/README.md). The console output at the end of the script will give you the information necessary to access and use the dev environment. Also, there is a video tutorial in Platform One IL2 Confluence. Search for "T3" and click the link to the page. Scroll down the page to the 57th video on 22-February-2022. There is a script that automates the creation and teardown of a remote k3d development environment. First, read the [script instructions](aws-k3d-script.md), understand what it does, and install required dependencies. Then, run the script [docs/assets/developer/scripts/k3d-dev.sh](../assets/scripts/developer/k3d-dev.sh) from your workstation. The console output at the end of the script will give you the information necessary to access and use the dev environment. Also, there is a video tutorial in Platform One IL2 Confluence. Search for "T3" and click the link to the page. Scroll down the page to the 57th video on 22-February-2022.
## Prerequisites ## Prerequisites
......
...@@ -134,7 +134,7 @@ spec: ...@@ -134,7 +134,7 @@ spec:
If Big Bang is deployed within AWS, KMS key access can be handled via IAM roles and permissions on the cluster resources themselves. If Big Bang is deployed within AWS, KMS key access can be handled via IAM roles and permissions on the cluster resources themselves.
However, if the deployment is in a different environment from the KMS keys, AWS credentials may need to be provided via a secret as follows. However, if the deployment is in a different environment from the KMS keys, AWS credentials may need to be provided via a secret as follows.
Configure the flux-system `kustomize-controller` component with AWS credential environment variables using `kustomize`. Specific instructions for doing this may vary by deployment and environment but [an example](https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/master#flux-components) is covered in the bigbang template repo. Broadly speaking, adding environment variables to the `kustomize-controller` component can be accomplished by adding a patch to the `flux/kustomization.yaml` for the target deployment or environment. An example of such a `kustomization.yaml` is shown below: Configure the flux-system `kustomize-controller` component with AWS credential environment variables using `kustomize`. Specific instructions for doing this may vary by deployment and environment but [an example](https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/main) is covered in the bigbang template repo. Broadly speaking, adding environment variables to the `kustomize-controller` component can be accomplished by adding a patch to the `flux/kustomization.yaml` for the target deployment or environment. An example of such a `kustomization.yaml` is shown below:
```yaml ```yaml
bases: bases:
......
...@@ -40,7 +40,7 @@ graph TD ...@@ -40,7 +40,7 @@ graph TD
## Values ## Values
Variables defined in Big Bang's [values.yaml](/chart/values.yaml) are values that the Big Bang team has identified as ones which users will be most likely to want to set when installing or upgrading Big Bang. This provides a single, standard way to set the most deployment-specific values and many users may not need to do any more than customize these values for their environment. Beyond these the Big Bang team also provides additional ways to pass values through to specific packages or modify templates after rendering. Variables defined in Big Bang's [values.yaml](../../../chart/values.yaml) are values that the Big Bang team has identified as ones which users will be most likely to want to set when installing or upgrading Big Bang. This provides a single, standard way to set the most deployment-specific values and many users may not need to do any more than customize these values for their environment. Beyond these the Big Bang team also provides additional ways to pass values through to specific packages or modify templates after rendering.
### Big Bang Configuration Values ### Big Bang Configuration Values
...@@ -102,4 +102,4 @@ kiali: ...@@ -102,4 +102,4 @@ kiali:
In some cases customers run into limitations that even values passthrough cannot solve. In cases where the package chart/template file itself does not provide a value for configuration you can make use of Post Renderers. In some cases customers run into limitations that even values passthrough cannot solve. In cases where the package chart/template file itself does not provide a value for configuration you can make use of Post Renderers.
These are an advanced capability within Helm and Flux that allows end users to make modifications to the chart after it has been rendered. For more information and specific on how to leverage this capability see [this document](/docs/postrenderers.md). These are an advanced capability within Helm and Flux that allows end users to make modifications to the chart after it has been rendered. For more information and specific on how to leverage this capability see [this document](../../postrenderers.md).
...@@ -60,4 +60,4 @@ Licensing of products deployable by BigBang are not covered by the BigBang team. ...@@ -60,4 +60,4 @@ Licensing of products deployable by BigBang are not covered by the BigBang team.
| Nexus | Generic Artifact Repository (AddOn App) | Nexus Repository OSS: Eclipse Public License v1.0 Nexus Repository Pro: Paid Licensed product | **Enterprise features of note:** HA, SAML SSO, Auth Token Support **Free tier notes:** A non-HA deployment can quickly auto heal thanks to Kubernetes, AWS S3 blob storage. <https://www.sonatype.com/products/repository-oss-vs-pro-features> <https://www.sonatype.com/products/pricing> | | Nexus | Generic Artifact Repository (AddOn App) | Nexus Repository OSS: Eclipse Public License v1.0 Nexus Repository Pro: Paid Licensed product | **Enterprise features of note:** HA, SAML SSO, Auth Token Support **Free tier notes:** A non-HA deployment can quickly auto heal thanks to Kubernetes, AWS S3 blob storage. <https://www.sonatype.com/products/repository-oss-vs-pro-features> <https://www.sonatype.com/products/pricing> |
| Gitlab, Gitlab Runners | GitRepo, Container Registry, and CICD Software Factory (AddOn App) | Gitlab Community Edition: MIT Expat license Gitlab Enterprise Edition: (multiple tiers) | **Premium features of note:** Release Controls, Project Management **Ultimate features of note:** Unlimited Guest Users, Advanced Security Testing (Note this functionality comes from container images that may not yet be in IronBank) **Free tier notes:** Free tier is fine for Proof of Concepts, but the Release Controls in Premium tier contain security controls that would be necessary for a cATO pipeline. PartyBus has multiple instances of Gitlab, most use Premium, a few use Ultimate. PartyBus's Gitlab pipelines integrate with additional licensed apps: Twistlock, Anchore, [Fortify](https://repo1.dso.mil/platform-one/big-bang/apps/third-party/fortify), [SD Elements](https://www.securitycompass.com/sdelements/), and others. (This is offered as a data point, it doesn't mean these are required for a cATO pipeline, the Consumer of BigBang's AO makes that call.) <https://about.gitlab.com/pricing/#self-managed> <https://gitlab.com/gitlab-org/gitlab-foss/-/tree/master#editions> | | Gitlab, Gitlab Runners | GitRepo, Container Registry, and CICD Software Factory (AddOn App) | Gitlab Community Edition: MIT Expat license Gitlab Enterprise Edition: (multiple tiers) | **Premium features of note:** Release Controls, Project Management **Ultimate features of note:** Unlimited Guest Users, Advanced Security Testing (Note this functionality comes from container images that may not yet be in IronBank) **Free tier notes:** Free tier is fine for Proof of Concepts, but the Release Controls in Premium tier contain security controls that would be necessary for a cATO pipeline. PartyBus has multiple instances of Gitlab, most use Premium, a few use Ultimate. PartyBus's Gitlab pipelines integrate with additional licensed apps: Twistlock, Anchore, [Fortify](https://repo1.dso.mil/platform-one/big-bang/apps/third-party/fortify), [SD Elements](https://www.securitycompass.com/sdelements/), and others. (This is offered as a data point, it doesn't mean these are required for a cATO pipeline, the Consumer of BigBang's AO makes that call.) <https://about.gitlab.com/pricing/#self-managed> <https://gitlab.com/gitlab-org/gitlab-foss/-/tree/master#editions> |
| SonarQube Community Edition | Static Code Analysis (AddOn App) | SonarQube CE: GNU Lesser GPL License v3 (Community Edition is Free/OSS) | An Enterprise Edition Exists, but is not bundled by BigBang | | SonarQube Community Edition | Static Code Analysis (AddOn App) | SonarQube CE: GNU Lesser GPL License v3 (Community Edition is Free/OSS) | An Enterprise Edition Exists, but is not bundled by BigBang |
| Anchore Enterprise Edition* | Vulnerability Scanner (AddOn App) | Anchore Enterprise Edition (Paid/Licensed) Anchore OpenSource Edition Apache License 2.0 (Free/OSS) | **Licensed features of note:** Proprietary Vulnerability Data Feeds for increased accuracy, NIST 800-190, Docker CIS Compliance, DoD container Policy Compliance, cATO Capable, RBAC, SSO **Free tier notes:** BigBang's values file can be set to deploy the OSS version for Proof of Concept deployments. PartyBus and other Platform One services use the licensed version <https://docs.anchore.com/current/docs/faq/#2> <https://anchore.com/pricing/> <https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/master/charter/packages/anchore/Architecture.md#licensing> <https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/anchore-enterprise/-/blob/main/docs/CHART.md#adding-enterprise-components> | | Anchore Enterprise Edition* | Vulnerability Scanner (AddOn App) | Anchore Enterprise Edition (Paid/Licensed) Anchore OpenSource Edition Apache License 2.0 (Free/OSS) | **Licensed features of note:** Proprietary Vulnerability Data Feeds for increased accuracy, NIST 800-190, Docker CIS Compliance, DoD container Policy Compliance, cATO Capable, RBAC, SSO **Free tier notes:** BigBang's values file can be set to deploy the OSS version for Proof of Concept deployments. PartyBus and other Platform One services use the licensed version <https://docs.anchore.com/3.0/docs/faq/#2> <https://anchore.com/pricing/> <https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/master/charter/packages/anchore/Architecture.md#licensing> <https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/anchore-enterprise/-/blob/main/docs/CHART.md#adding-enterprise-components> |
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment