diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 281d39ec9903fd2515295c9141770e7f9f23cf63..ce72183c661cc6d0b72d9a9e8f388daabd4ee919 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,12 +6,6 @@ Table of Contents: - [Contributing to Big Bang](#contributing-to-big-bang) - [Developers Guide](#developers-guide) - - [Local Git Setup](#local-git-setup) - - [Pre-commit hooks](#pre-commit-hooks) - - [Local Setup](#local-setup) - - [Prereqs](#prereqs) - - [Steps](#steps) - - [Combining Multiple Commits](#combining-multiple-commits) - [Iron Bank Images](#iron-bank-images) - [Local Kubernetes cluster](#local-kubernetes-cluster) - [Deploying Big Bang (Quick Start)](#deploying-big-bang-quick-start) diff --git a/charter/BigBangPackages.md b/charter/BigBangPackages.md index bf08093e781945902e2fcf060eff30f5040b3a3f..01e5d711f9ad03270506cb9aa01cba939768ed4b 100644 --- a/charter/BigBangPackages.md +++ b/charter/BigBangPackages.md @@ -218,7 +218,7 @@ Repository: [Cluster Auditor](https://repo1.dso.mil/platform-one/big-bang/apps/c Dependencies: -* [Logging](#Logging) +* [Logging](#logging) * [OPA Gatekeer](#policy-enforcement) Owners: diff --git a/charter/DocumentationLocations.md b/charter/DocumentationLocations.md index e14a4a16bc4be2f8d8a7e839494973173e0213c6..1cb4a5495eafcf9555721297ee3ecfee004649ac 100644 --- a/charter/DocumentationLocations.md +++ b/charter/DocumentationLocations.md @@ -23,4 +23,5 @@ - Our goal is to have these docs available on a webpage hosted on the BigBang Cluster using Hugo (https://docs.bigbang.dev by default) (look [here](./PackageDocumentation.md) for more info) +- Currently the docs are hosted for consumption on https://docs-bigbang.dso.mil/ - This allows us to support a centralized location for package configuration documentation, while allowing support for dynamically added versions of distributed packages in a maintainable way. diff --git a/charter/GitOpsEngine.md b/charter/GitOpsEngine.md index cc6d135b6e7098b3f5c2ae530ead8f40e509e5ee..e159bd74a7dcb3004ba2d49318f9b85e23de59db 100644 --- a/charter/GitOpsEngine.md +++ b/charter/GitOpsEngine.md @@ -16,8 +16,8 @@ The Flux2 Engine has native Helm support, meaning the controller deployed as par Argo, has taken the ownership of rendering and managing the lifecycle of applications that does not work exactly as expected by helm. As a result, there are several vendor Helm Charts that **do not deploy successfully** with Argo because of how Argo shims Helm Hooks to Argo specific sync phases. -* GitLab initial secret creation is performed via a [subchart]([https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/charts/shared-secrets](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/charts/shared-secrets)) -* Kube Prometheus Stack - [prometheusrule admission webhook]([https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusrules-admission-webhooks](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusrules-admission-webhooks)) is created via a helm `install` hook +* GitLab initial secret creation is performed via a [subchart](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/charts/shared-secrets) +* Kube Prometheus Stack - [prometheusrule admission webhook](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusrules-admission-webhooks) is created via a helm `install` hook * Confluent - A deletion hook is part of a subchart gets run at different point in the lifecycle As new features of Helm get developed and leveraged by the community, we would need to lean on the time and availability of the Argo developers to re-implement the capabilities. diff --git a/charter/packages/anchore/Architecture.md b/charter/packages/anchore/Architecture.md index 89fd63bc0b6575d3f9e3d2f68fa16abd7417a0b2..abdb76dd73a0554e800fbb5556e5d16a887cfd50 100644 --- a/charter/packages/anchore/Architecture.md +++ b/charter/packages/anchore/Architecture.md @@ -90,7 +90,7 @@ addons: ### Single Sign On -Anchore Enterprise 2.1+ can be configured to support user login to the UI using identities from external identity providers that support SAML 2.0. In such a configuration, Anchore never stores any credentials for the users, only their usernames and Anchore permissions, and all UI access is gated through a user’s valid login into the identity provider. Anchore uses the external provider to verify username identity and initialize a username, account, and roles on first login for a new user. Once a user’s identity is initialized in Anchore, the Anchore administrator may manage user permissions by managing the roles associated with the user’s identity in Anchore itself. For more information, see [Anchore Enterprise SSO Support](https://docs.anchore.com/current/docs/overview/sso/). +Anchore Enterprise 2.1+ can be configured to support user login to the UI using identities from external identity providers that support SAML 2.0. In such a configuration, Anchore never stores any credentials for the users, only their usernames and Anchore permissions, and all UI access is gated through a user’s valid login into the identity provider. Anchore uses the external provider to verify username identity and initialize a username, account, and roles on first login for a new user. Once a user’s identity is initialized in Anchore, the Anchore administrator may manage user permissions by managing the roles associated with the user’s identity in Anchore itself. For more information, see [Anchore Enterprise SSO Support](https://docs.anchore.com/current/docs/configuration/sso/). See below for an example of the values to provide to Anchore Enterprise for SSO setup: @@ -105,7 +105,7 @@ addons: ### Storage -Anchore relies on a PostgreSQL database as its primary data store. By default, Anchore will deploy an in-cluster PostgreSQL database, but it is recommended that an external PostgreSQL 9.6+ database be used, which can be configured in the Big Bang values.yaml. For more information, see [Anchore Enterprise Storage Overview](https://docs.anchore.com/current/docs/installation/storage/). +Anchore relies on a PostgreSQL database as its primary data store. By default, Anchore will deploy an in-cluster PostgreSQL database, but it is recommended that an external PostgreSQL 9.6+ database be used, which can be configured in the Big Bang values.yaml. For more information, see [Anchore Enterprise Storage Overview](https://docs.anchore.com/current/docs/configuration/storage/). ### High Availability @@ -135,7 +135,7 @@ _Note:_ within Big Bang, logs are captured by fluentbit and shipped to elastic b ### Monitoring -Anchore Engine and Enterprise expose prometheus metrics in the API of each service if the config.yaml used by that service has the metrics.enabled key set to true. Each service exports its own metrics and is typically scraped by a Prometheus installation to gather the metrics. Anchore does not aggregate or distribute metrics between services. You should configure your Prometheus deployment or integration to check each Anchore service’s api (using the same port it exports), for the /metrics route. For more information, see [Anchore Enterprise Monitoring](https://docs.anchore.com/current/docs/monitoring/#monitoring-in-kubernetes-andor-helm-chart). +Anchore Engine and Enterprise expose prometheus metrics in the API of each service if the config.yaml used by that service has the metrics.enabled key set to true. Each service exports its own metrics and is typically scraped by a Prometheus installation to gather the metrics. Anchore does not aggregate or distribute metrics between services. You should configure your Prometheus deployment or integration to check each Anchore service’s api (using the same port it exports), for the /metrics route. For more information, see [Anchore Enterprise Monitoring](https://docs.anchore.com/current/docs/configuration/monitoring/#monitoring-in-kubernetes-andor-helm-chart). The Big Bang Anchore Helm chart has been modified to use your `monitoring:` values in Big Bang to automatically toggle metrics on/off. diff --git a/charter/packages/minio/Architecture.md b/charter/packages/minio/Architecture.md index e4fda223226657e355cc4d04557b95d740f46fa1..7ca08a776b16d73bea3338564e4b4dcd63d38e68 100644 --- a/charter/packages/minio/Architecture.md +++ b/charter/packages/minio/Architecture.md @@ -13,7 +13,7 @@ The MinIO tenants are created using the [MinIO package](https://repo1.dso.mil/pl The final package is the MinIO console. This is a graphical user interface that allows management of an individual tenant. The official package can be found [here](https://repo1.dso.mil/platform-one/big-bang/apps/application-utilities/minio). - +[Tenant Architecture](https://raw.githubusercontent.com/minio/operator/master/docs/images/architecture.png) Note: The Minio Operator needs to be able to reach out to the minio instances. This is to ensure that on an upgrade all existing pools are shut down before starting new ones. If you run into issues with upgrades ensure that networkPolicies allow ingress to the minio pods in your namespace on port 9000. diff --git a/docs/README.md b/docs/README.md index 961b50d043e28dd41ff26427d8d3bdab946ae271..ce3ed896c2be6679c904a9b5c4c9ca900f191ff1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -32,4 +32,4 @@ The following is a general overview of the process, the [deployment guides](guid ## New User Orientation -* New users are encouraged to read through the Useful Background Contextual Information present in the [understanding_bigbang folder](./understanding_bigbang) +* New users are encouraged to read through the Useful Background Contextual Information present in the [understanding-bigbang folder](./understanding-bigbang) diff --git a/docs/airgap/README.md b/docs/airgap/README.md index 89fe0a73d41fe0f470da38f0d2488a08af460342..8ac046b31b459af52146c62cd4581a664cd4b1a1 100644 --- a/docs/airgap/README.md +++ b/docs/airgap/README.md @@ -252,7 +252,7 @@ If you need to handle mirroring manually, there is an example Ansible script pro #### Konvoy Cluster -Modify the `cluster.yaml` file and apply. More details can be found on the [D2iQ Konvoy documentation](https://docs.d2iq.com/dkp/konvoy/1.6/install/install-airgapped/). +Modify the `cluster.yaml` file and apply. More details can be found on the [D2iQ Konvoy documentation](https://docs.d2iq.com/dkp/konvoy/1.8/install/install-airgapped/). ```yaml kind: ClusterConfiguration diff --git a/docs/developer/aws-k3d-script.md b/docs/developer/aws-k3d-script.md index 5a105adfba9f9d02134e0ba8cee64f25a2153da2..3f819f48043385ecf5a8f7a183a5b225eb86c125 100644 --- a/docs/developer/aws-k3d-script.md +++ b/docs/developer/aws-k3d-script.md @@ -62,7 +62,8 @@ Follow the instructions from the script output to access and use the cluster. ## Install FluxCD -The Big Bang product is tightly coupled with the GitOps tool FluxCD. Before you can deploy Big Bang you must deploy FluxCD on your k8s cluster. To guarantee that you are using the version of FluxCD that is compatible with the version of Big Bang that you are deploying use the Big Bang provided [script](./scripts/install_flux.sh). You will need your Iron Bank pull credentials and command line access to the k8s cluster from your workstation. +The Big Bang product is tightly coupled with the GitOps tool FluxCD. Before you can deploy Big Bang you must deploy FluxCD on your k8s cluster. To guarantee that you are using the version of FluxCD that is compatible with the version of Big Bang that you are deploying use the Big Bang provided [script](../../scripts/install_flux.sh). You will need your Iron Bank pull credentials and command line access to the k8s cluster from your workstation. + ```shell ./scripts/install_flux.sh -u your-user-name -p your-password ``` diff --git a/docs/developer/development-environment.md b/docs/developer/development-environment.md index eac4b57aa6a221592c4b647a913decc3933dbbc3..2a16d1cfeee7920ea1cbc6d0590be7e564da099a 100644 --- a/docs/developer/development-environment.md +++ b/docs/developer/development-environment.md @@ -4,9 +4,9 @@ BigBang developers use [k3d](https://k3d.io/), a lightweight wrapper to run [k3s](https://github.com/rancher/k3s) (Rancher Lab’s minimal Kubernetes distribution) in Docker. K3d is a virtualized kubernetes cluster that is quick to start and tear down for fast development iteration. K3d is sufficient for 95% of BigBang development work. In limited cases developers will use real infrastructure k8s deployments with Rancher, Konvoy, EKS, etc. Only k3d is covered in this document. -It is not recommend to run k3d with Big Bang on your local computer. Instead use a remote k3d cluster running on an EC2 instance to shift the compute and network bandwidth to the cloud. Big Bang can be quite resource intensive and it requires a huge download bandwidth for the images. If you do insist on running k3d locally you should disable certain packages before deploying. You can do this in the values.yaml file by setting the package deploy to false. One of the packages that is most resource-intensive is the logging package. And you should create a local image registry cache to minimize the amount of image downloading. +It is not recommend to run k3d with Big Bang on your local workstation. Instead use a remote k3d cluster running on an EC2 instance to shift the compute and network bandwidth to the cloud. Big Bang can be quite resource intensive and it requires a huge download bandwidth for the images. If you do insist on running k3d locally you should disable certain packages before deploying. You can do this in the values.yaml file by setting the package deploy to false. One of the packages that is most resource-intensive is the logging package. And you should create a local image registry cache to minimize the amount of image downloading. -There is a script [/docs/developer/scripts/k3d-dev.sh](./scripts/) that automates the creation and teardown of a remote k3d development environment. Run the script from your workstation after [installing dependencies](/docs/developer/scripts/README.md). The console output at the end of the script will give you the information necessary to access and use the dev environment. Also, there is a video tutorial in Platform One IL2 Confluence. Search for "T3" and click the link to the page. Scroll down the page to the 57th video on 22-February-2022. +There is a script that automates the creation and teardown of a remote k3d development environment. First, read the [script instructions](aws-k3d-script.md), understand what it does, and install required dependencies. Then, run the script [docs/assets/developer/scripts/k3d-dev.sh](../assets/scripts/developer/k3d-dev.sh) from your workstation. The console output at the end of the script will give you the information necessary to access and use the dev environment. Also, there is a video tutorial in Platform One IL2 Confluence. Search for "T3" and click the link to the page. Scroll down the page to the 57th video on 22-February-2022. ## Prerequisites diff --git a/docs/encryption.md b/docs/encryption.md index 64ca23d2802d5551395a32a905d950f961d44896..6a53ce01e333fd7c6b1ddb58b08efb7f49256306 100644 --- a/docs/encryption.md +++ b/docs/encryption.md @@ -134,7 +134,7 @@ spec: If Big Bang is deployed within AWS, KMS key access can be handled via IAM roles and permissions on the cluster resources themselves. However, if the deployment is in a different environment from the KMS keys, AWS credentials may need to be provided via a secret as follows. -Configure the flux-system `kustomize-controller` component with AWS credential environment variables using `kustomize`. Specific instructions for doing this may vary by deployment and environment but [an example](https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/master#flux-components) is covered in the bigbang template repo. Broadly speaking, adding environment variables to the `kustomize-controller` component can be accomplished by adding a patch to the `flux/kustomization.yaml` for the target deployment or environment. An example of such a `kustomization.yaml` is shown below: +Configure the flux-system `kustomize-controller` component with AWS credential environment variables using `kustomize`. Specific instructions for doing this may vary by deployment and environment but [an example](https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/main) is covered in the bigbang template repo. Broadly speaking, adding environment variables to the `kustomize-controller` component can be accomplished by adding a patch to the `flux/kustomization.yaml` for the target deployment or environment. An example of such a `kustomization.yaml` is shown below: ```yaml bases: diff --git a/docs/guides/using-bigbang/values-guide.md b/docs/guides/using-bigbang/values-guide.md index 25302ef0534d9e664184271d0af185aff4ce9af2..c734edc51f53c8034478f09a856b90208b40062f 100644 --- a/docs/guides/using-bigbang/values-guide.md +++ b/docs/guides/using-bigbang/values-guide.md @@ -40,7 +40,7 @@ graph TD ## Values -Variables defined in Big Bang's [values.yaml](/chart/values.yaml) are values that the Big Bang team has identified as ones which users will be most likely to want to set when installing or upgrading Big Bang. This provides a single, standard way to set the most deployment-specific values and many users may not need to do any more than customize these values for their environment. Beyond these the Big Bang team also provides additional ways to pass values through to specific packages or modify templates after rendering. +Variables defined in Big Bang's [values.yaml](../../../chart/values.yaml) are values that the Big Bang team has identified as ones which users will be most likely to want to set when installing or upgrading Big Bang. This provides a single, standard way to set the most deployment-specific values and many users may not need to do any more than customize these values for their environment. Beyond these the Big Bang team also provides additional ways to pass values through to specific packages or modify templates after rendering. ### Big Bang Configuration Values @@ -102,4 +102,4 @@ kiali: In some cases customers run into limitations that even values passthrough cannot solve. In cases where the package chart/template file itself does not provide a value for configuration you can make use of Post Renderers. -These are an advanced capability within Helm and Flux that allows end users to make modifications to the chart after it has been rendered. For more information and specific on how to leverage this capability see [this document](/docs/postrenderers.md). +These are an advanced capability within Helm and Flux that allows end users to make modifications to the chart after it has been rendered. For more information and specific on how to leverage this capability see [this document](../../postrenderers.md). diff --git a/docs/understanding-bigbang/licensing-expectations.md b/docs/understanding-bigbang/licensing-expectations.md index ca331e600493304467fed265256fca9e45459725..7e5d038e6683ab2911daf7cfd7094dacbad23eac 100644 --- a/docs/understanding-bigbang/licensing-expectations.md +++ b/docs/understanding-bigbang/licensing-expectations.md @@ -60,4 +60,4 @@ Licensing of products deployable by BigBang are not covered by the BigBang team. | Nexus | Generic Artifact Repository (AddOn App) | Nexus Repository OSS: Eclipse Public License v1.0 Nexus Repository Pro: Paid Licensed product | **Enterprise features of note:** HA, SAML SSO, Auth Token Support **Free tier notes:** A non-HA deployment can quickly auto heal thanks to Kubernetes, AWS S3 blob storage. <https://www.sonatype.com/products/repository-oss-vs-pro-features> <https://www.sonatype.com/products/pricing> | | Gitlab, Gitlab Runners | GitRepo, Container Registry, and CICD Software Factory (AddOn App) | Gitlab Community Edition: MIT Expat license Gitlab Enterprise Edition: (multiple tiers) | **Premium features of note:** Release Controls, Project Management **Ultimate features of note:** Unlimited Guest Users, Advanced Security Testing (Note this functionality comes from container images that may not yet be in IronBank) **Free tier notes:** Free tier is fine for Proof of Concepts, but the Release Controls in Premium tier contain security controls that would be necessary for a cATO pipeline. PartyBus has multiple instances of Gitlab, most use Premium, a few use Ultimate. PartyBus's Gitlab pipelines integrate with additional licensed apps: Twistlock, Anchore, [Fortify](https://repo1.dso.mil/platform-one/big-bang/apps/third-party/fortify), [SD Elements](https://www.securitycompass.com/sdelements/), and others. (This is offered as a data point, it doesn't mean these are required for a cATO pipeline, the Consumer of BigBang's AO makes that call.) <https://about.gitlab.com/pricing/#self-managed> <https://gitlab.com/gitlab-org/gitlab-foss/-/tree/master#editions> | | SonarQube Community Edition | Static Code Analysis (AddOn App) | SonarQube CE: GNU Lesser GPL License v3 (Community Edition is Free/OSS) | An Enterprise Edition Exists, but is not bundled by BigBang | -| Anchore Enterprise Edition* | Vulnerability Scanner (AddOn App) | Anchore Enterprise Edition (Paid/Licensed) Anchore OpenSource Edition Apache License 2.0 (Free/OSS) | **Licensed features of note:** Proprietary Vulnerability Data Feeds for increased accuracy, NIST 800-190, Docker CIS Compliance, DoD container Policy Compliance, cATO Capable, RBAC, SSO **Free tier notes:** BigBang's values file can be set to deploy the OSS version for Proof of Concept deployments. PartyBus and other Platform One services use the licensed version <https://docs.anchore.com/current/docs/faq/#2> <https://anchore.com/pricing/> <https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/master/charter/packages/anchore/Architecture.md#licensing> <https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/anchore-enterprise/-/blob/main/docs/CHART.md#adding-enterprise-components> | +| Anchore Enterprise Edition* | Vulnerability Scanner (AddOn App) | Anchore Enterprise Edition (Paid/Licensed) Anchore OpenSource Edition Apache License 2.0 (Free/OSS) | **Licensed features of note:** Proprietary Vulnerability Data Feeds for increased accuracy, NIST 800-190, Docker CIS Compliance, DoD container Policy Compliance, cATO Capable, RBAC, SSO **Free tier notes:** BigBang's values file can be set to deploy the OSS version for Proof of Concept deployments. PartyBus and other Platform One services use the licensed version <https://docs.anchore.com/3.0/docs/faq/#2> <https://anchore.com/pricing/> <https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/master/charter/packages/anchore/Architecture.md#licensing> <https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/anchore-enterprise/-/blob/main/docs/CHART.md#adding-enterprise-components> |