diff --git a/charter/BigBangPackages.md b/charter/BigBangPackages.md index 2b85f2f6c30abee2d25e2af085e8fa53b7000eba..8ff064d68a57b538bdc8bb71753b18efef5e0394 100644 --- a/charter/BigBangPackages.md +++ b/charter/BigBangPackages.md @@ -61,7 +61,6 @@ graph TB ``` - ## Core Core packages are supported Big Bang packages that have to be enabled and are located at [Big Bang Core](https://repo1.dso.mil/platform-one/big-bang/apps/core). Core packages are platform/admin level packages that are leveraged by other packages. @@ -110,7 +109,7 @@ Owners: Understudy: -* @kavitha +* @kavitha ### Service Mesh @@ -150,7 +149,6 @@ Repository: * [authservice](https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/authservice) - Dependency: None Owners: @@ -173,7 +171,8 @@ The logging capability is comprised of: * Fluentd * Logging Operator -Repository: +Repository: + * [Elasticsearch-kibana](https://repo1.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana) * [Fluentbit](https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit) * [Eck-operator](https://repo1.dso.mil/platform-one/big-bang/apps/core/eck-operator) @@ -260,6 +259,7 @@ Understudy: * @kenna81 Repository: + * [Cluster Auditor Repo](https://repo1.dso.mil/platform-one/big-bang/apps/core/cluster-auditor) ### Twistlock @@ -282,6 +282,7 @@ Owners: * @thomas.burton - iSenpai ## Addons + Addons are supported Big Bang packages that come disabled by default. ### Security Tools @@ -412,6 +413,7 @@ Owners: * @LynnStill Understudies + * @kevin.wilder #### Sonarqube @@ -438,11 +440,11 @@ Owners: #### Fortify -Fortify provides code +Fortify provides code Product: -* +* Repository: @@ -462,7 +464,7 @@ Nexus provides a robust artifact repository, supporting artifacts of multiple pr Product: * [Nexus](https://www.sonatype.com/nexus/repository-pro) -* Scope: +* Scope: * The Nexus OSS will not be supported as the licenced pro version is required for [HA and SAML SSO capabilities](https://www.sonatype.com/nexus/repository-oss-vs-pro-features) * Only Licended Nexus Repository Pro will be supported @@ -519,7 +521,6 @@ Owners: * @matt.kaiser * @branden.cobb - #### Jira Development tool for planning and tracking team tasks diff --git a/charter/GitlabLabels.md b/charter/GitlabLabels.md index 2e1015c2708d342a2d0e51a82a7390416397d792..5124e050bf4a0e4611b3031ef79cc78837cec2e5 100644 --- a/charter/GitlabLabels.md +++ b/charter/GitlabLabels.md @@ -13,7 +13,7 @@ The kind label shows the type of work that needs to be accomplished #### `kind::bug` -Issues releated to Bigbang not functioning as expected +Issues related to Bigbang not functioning as expected #### `kind::chore` @@ -21,11 +21,11 @@ Catch all kind that captures administrative tasking for the BigBang project #### `kind:ci` -Issues related to the CI/CD, developer workflows and/or the releaes process +Issues related to the CI/CD, developer workflows and/or the release process #### `kind::docs` -Issues related to documentaiton +Issues related to documentation #### `kind::feature` @@ -33,7 +33,7 @@ Creation of a new capability for BigBang and/or one of its packages #### `kind::enhancement` -Improvement of an existing capablity to work more efficiently in specific environments +Improvement of an existing capability to work more efficiently in specific environments #### `kind::test` @@ -43,7 +43,7 @@ Improvements on testing for individual packages or Big Bang. Does not change th #### `priority::high` -`priority::high` issues are causing runtime issues in production enviornments. These issues justify a patch of a release. +`priority::high` issues are causing runtime issues in production environments. These issues justify a patch of a release. #### `priority:: medium` @@ -75,7 +75,7 @@ This Issue has not been started. ### Packages -Package labels are identified by their package name and serve two purposes. +Package labels are identified by their package name and serve two purposes. 1. Packages owners subscribe to the package labels for their packages and will be notified when a new issue or merge request is created with the label @@ -163,7 +163,7 @@ The `size` label helps identify the scope of work needed as part of the epic #### `size::small` -Sufficently small enough to be completed by an engineer in a two week period +Sufficiently small enough to be completed by an engineer in a two week period #### `size::medium` diff --git a/charter/New GitLab Epic Checklist Template b/charter/New GitLab Epic Checklist Template index 6f57ac51ec4410db1901c8cfac09958ea8a1b74e..b66da164537a340a373e83ccca72f6437ea4d505 100644 --- a/charter/New GitLab Epic Checklist Template +++ b/charter/New GitLab Epic Checklist Template @@ -28,7 +28,7 @@ not used. - [ ] Does the upstream chart version deploy the application version used in Iron Bank (or as close as possible)? This will help avoid incompatible configuration settings. -- [ ] Have you run helm dep up and added all .tgz file depedencies in chart/charts to the repo? +- [ ] Have you run helm dep up and added all .tgz file dependencies in chart/charts to the repo? - [ ] Have you updated chart/requirements.yaml or chart/Chart.yaml to point to the file://./charts/<file>.tgz dependencies? - [ ] If the chart has a web interface, have you added a VirtualService using hostname that is conditionally added if istio.enabled is true? Verify this works using the web address. @@ -40,7 +40,7 @@ pulling from registry.dso.mil? - [ ] If the package supports SSO, have you integrated SSO settings? Needs clarification - [ ] If the package requires a database, have you integrated external database settings? Needs clarification -- [ ] If the package requires storage, have you ingegrated external storage (e.g. MinIO) settings? Needs clarification +- [ ] If the package requires storage, have you integrated external storage (e.g. MinIO) settings? Needs clarification - [ ] Are all secrets and certificates removed from the repo? All secrets should be references or randomly generated during deployment. @@ -71,7 +71,7 @@ Additional details on definition of done need to be added for: - Database integration -- Storage (minio) ingegration +- Storage (minio) integration - Certificates? - SSO integration diff --git a/charter/PackageDocumentation.md b/charter/PackageDocumentation.md index 660858d4cee7038e634a779220eddeb588a03ebe..85b8c519569547577896913e74bb2a54044829f7 100644 --- a/charter/PackageDocumentation.md +++ b/charter/PackageDocumentation.md @@ -52,7 +52,7 @@ package | index.html [landing page for application within hugo] | overview.md [Overview of application, purpose, and default config options] | keycloak.md [Manual or automated steps to configure sso] - | example-optional-config.md [Addtional files for optional configuration for the app] + | example-optional-config.md [Additional files for optional configuration for the app] ``` ### Formatting Needed diff --git a/charter/PackageOwner.md b/charter/PackageOwner.md index a04d5dfcedef74ef5aff7c3d8cb6905357f47f8f..77d36ce64b89774a1fc2f154d220bda669b53a74 100644 --- a/charter/PackageOwner.md +++ b/charter/PackageOwner.md @@ -6,14 +6,14 @@ Package owners will be responsible for the following: * Implementing Package requirements outlined by [Package Requirements](PackageRequirements.md) * Reviewing Merge Requests into the Package Repository * Reviewing Merge Request CI/CD pipeline execution results to ensure that there are no regressions in conformance tests nor package cypress tests. -* Tracking upstream changes to packages including new features, architectures, depedenecies. +* Tracking upstream changes to packages including new features, architectures, dependencies. * Upgrading package with new upstream versions * Implementing features based on customer requests/requirements * Adding and improving interactions with current new new Big Bang Packages * IronBank interactions * Identifying new Images to harden * Notify IronBank of new versions available - * Testing new IronBank imaages + * Testing new IronBank images * [Long term] Providing CI processes for hardening images Package Owners will be identified by the use of [CODEOWNERS](https://docs.gitlab.com/ee/user/project/code_owners.html) files in the repository. diff --git a/charter/PackageRequirements.md b/charter/PackageRequirements.md index 2493a1c490c52e41bc1e45ec0e60d3a18c3b3567..baf227a444d1e403b75f6dd6fdc8af396a12e4de 100644 --- a/charter/PackageRequirements.md +++ b/charter/PackageRequirements.md @@ -9,8 +9,8 @@ All Big Bang Packages shall adhere to the following requirements. Where possibl * The `main` branch should be default in each project * Merge Requests should require 1 approver * The `main` branch should be protected: - * Devs + Maintainers should be allowed to merge. - * No one should be allowed to push and it should allow + * Developers + Maintainers should be allowed to merge. + * No one should be allowed to push and it should allow * CODEOWNERs approval should be allowed * There should exist a protected tag with the wildcard `*-bb*` @@ -47,7 +47,7 @@ There are two types of third party packages: These are packages that are supported, updated, and maintained by team members of BigBang. This designation is usually reserved for packages that key customers require, but are missing approved IronBank containers, or blanket approval that allows them to be included with the BigBang product. -These products are labeled with the "BigBang Supported" badge on the repositorys `README.md` page, which indicates active support. That being said, BigBang reserves the right to deprecate support for these packages. +These products are labeled with the "BigBang Supported" badge on the repository's `README.md` page, which indicates active support. That being said, BigBang reserves the right to deprecate support for these packages. #### Independent @@ -85,7 +85,7 @@ For another example in using the [`kube-prometheus-stack`](https://github.com/pr The common components that each package will have are defined in the following folder layout: -```bash +```shell ├── CODEOWNERS # GitLab Code Owners for Package Owners/Understudies. ├── README.md # top level summary of package ├── docs/ # detailed documentation folder describing package consumption details and assumptions @@ -101,7 +101,7 @@ The common components that each package will have are defined in the following f Each package shall contain a .gitlab-ci.yml file at the top of the package repo. This file shall reference the pipeline CI/CD infrastructure files and include the following contents: -```bash +```shell include: - project: 'platform-one/big-bang/pipeline-templates/pipeline-templates' ref: master @@ -159,5 +159,3 @@ Each package will have a default branch of `main`. Immutable tags will be used version: ~1.2.3 * There should be a Helm values file located at `tests/test-values.yaml` used for pipeline testing. * Charts should support `affinity` and `nodeSelector` configuration for all components. If there is only one type of `Pods`, then a single, top level value shall be provided, otherwise there should be `affinity` and `nodeSelector` regions for each component. See [the Kubernetes Docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) for more information - - diff --git a/charter/Scratch.md b/charter/Scratch.md index f6aea654f9b79f0d845fd56b2ad47c24198ea92d..df226393b2d08ef65095b36a852275ede2ed535a 100644 --- a/charter/Scratch.md +++ b/charter/Scratch.md @@ -1,39 +1,29 @@ -# +# Scratch + Comments from meetings ## Apps -* Should any product need to be licenced (e.g. why Anchore Enterprise and not Anchore) -* Should everything used by apps be internal? E.g. Postgres requried for Keycloak +* Should any product need to be licence (e.g. why Anchore Enterprise and not Anchore) +* Should everything used by apps be internal? E.g. Postgres required for Keycloak * Which of Anchore vs Anchore Enterprise * Consistent Interface. Only "supported" BigBang configuration - - - - Testing stuff - - - - -## Kubernetes Tools E2E Testing Frameworks: +## Kubernetes Tools E2E Testing Frameworks ### Each Applications E2E tests -Is there a way to get each applicatoin to run its own e2e tests against the deployed version? +Is there a way to get each application to run its own e2e tests against the deployed version? e.g. for Argo: -https://github.com/argoproj/argo-cd/blob/master/.github/workflows/ci-build.yaml - - - +<https://github.com/argoproj/argo-cd/blob/master/.github/workflows/ci-build.yaml> ### Istio -Istio uses Prow: https://github.com/istio/test-infra +Istio uses Prow: <https://github.com/istio/test-infra> ### KUTTL @@ -41,44 +31,32 @@ KUTTL allows for the verification of Kubernetes objects (and status) based on ap This easily allows for testing the health of all the objects (per status fields), but doesn't provide integration tests unless we build all the integration tests into CRDs or into Kubernetes Jobs: - - APP -* manfiests/linting + +* manifests/linting * k3d healthy" * smoke tests - - - Integration Tests -* - - +* Single release of all app versions in single place. Tested by BB Customer extensions need to be tested in their own moc environment - - Common Integration: -* "App of Apps" - - +* "App of Apps" Mock Integration environments -* sample implementation of customer - +* sample implementation of customer -# Keycloak +## Keycloak Table discussion - - API:* + * can't change image tags -* can change repo to allow for airgapped repos \ No newline at end of file +* can change repo to allow for airgapped repos diff --git a/charter/Testing.md b/charter/Testing.md index d8541b3e544ebcf3a50d54642219d57144c9c830..063b6ff76ca34a6577ef848d4d1d5c478b09d0b5 100644 --- a/charter/Testing.md +++ b/charter/Testing.md @@ -57,10 +57,10 @@ In order to add Helm Chart tests to your application, the following enhancements * A test directory is added to templates/ directory within the helm chart. This directory contains Kubernetes object definitions which are deployed only when a "helm Test" command is executed. As an example, tests can be YAML files that -execute pods with containers, deply config maps, secrets, or other objects. -* When a files contain a pod / container defintion that executions tests, the container must return success or failure +execute pods with containers, deploy config maps, secrets, or other objects. +* When a files contain a pod / container definition that executions tests, the container must return success or failure (i.e., the container should exit successfully with an exit 0 for a test to be considered a success). -* Each test object defintion must contain a "helm.sh/hook: test-success" annotation telling Helm that this object is a +* Each test object definition must contain a "helm.sh/hook: test-success" annotation telling Helm that this object is a test and should only be deployed when tests are executed. The following example create a configmap that is only created during testing. @@ -81,6 +81,7 @@ data: {{ (.Files.Glob "cypress/*").AsConfig | indent 2 }} ``` + Also note that the "helm.sh/hook-weight" can be used to order the creation and execution of test objects. ## Umbrella Testing diff --git a/charter/packages/README.md b/charter/packages/README.md index c8bd8474d45381ba64616ae5c113f89ceb792762..dc8415ea9f4cdc2b503eb6c9f0f4982721a64601 100644 --- a/charter/packages/README.md +++ b/charter/packages/README.md @@ -1,3 +1,5 @@ -# All packages should have a folder under the /packages directory named /packages/<name> that contains all relevant information relted to the Big Bang team assessment for inclusion into the Big Bang baseline. +# README -# Do we maintain this information as packages are approved? +All packages should have a folder under the `/packages` directory named `/packages/<name>` that contains all relevant information related to the Big Bang team assessment for inclusion into the Big Bang baseline + +**Do we maintain this information as packages are approved?** diff --git a/charter/packages/anchore/Architecture.md b/charter/packages/anchore/Architecture.md index abbbc5d966915e06a02f614a1d69609dac6ec9f0..7b809fb3e64f809b43dba423f0530481fe0ea3c7 100644 --- a/charter/packages/anchore/Architecture.md +++ b/charter/packages/anchore/Architecture.md @@ -4,9 +4,10 @@ [Anchore](https://anchore.com/) is a Docker container static analysis and policy-based compliance system that automates the inspection, analysis, and evaluation of images against user-defined checks to allow high confidence in container deployments by ensuring workload content meets the required criteria. -Anchore offers several [open source tools](https://anchore.com/opensource/) and products, however, this document will cover the architectural touchpoints for the Big Bang Anchore package, which includes Anchore Engine (open source) and Anchore Enterprise (requires enterprise license). For more information on the differentiators between Anchore's open source and commercial offerings, see [here](https://anchore.com/pricing/). +Anchore offers several [open source tools](https://anchore.com/opensource/) and products, however, this document will cover the architectural touch points for the Big Bang Anchore package, which includes Anchore Engine (open source) and Anchore Enterprise (requires enterprise license). For more information on the differentiators between Anchore's open source and commercial offerings, see [here](https://anchore.com/pricing/). ### Anchore Engine + ```mermaid graph LR subgraph "Anchore Engine" @@ -28,6 +29,7 @@ graph LR ``` ### Anchore Enterprise + ```mermaid graph LR subgraph "Anchore Enterprise" @@ -67,15 +69,16 @@ graph LR For more information on the Anchore Enterprise architecture, see [Enterprise Service Overview and Architecture](https://docs.anchore.com/current/docs/overview/architecture/). -## Big Bang Touchpoints +## Big Bang Touch Points ### Licensing -The Big Bang Anchore Enterprise services require a valid Anchore Enterprise license as well as credentials with access to Registry1 hosting the hardened images. +The Big Bang Anchore Enterprise services require a valid Anchore Enterprise license as well as credentials with access to Registry1 hosting the hardened images. To be onboarded and provided with a trial or production license, please send an email to publicsector@anchore.com including program name and contact details. Once you have obtained a license this can be added to your values in Big Bang to automatically set up your Anchore deployment with the license (replacing the `licenseYaml:` value with your full license): + ```yaml addons: anchore: @@ -90,6 +93,7 @@ addons: Anchore Enterprise 2.1+ can be configured to support user login to the UI using identities from external identity providers that support SAML 2.0. In such a configuration, Anchore never stores any credentials for the users, only their usernames and Anchore permissions, and all UI access is gated through a user’s valid login into the identity provider. Anchore uses the external provider to verify username identity and initialize a username, account, and roles on first login for a new user. Once a user’s identity is initialized in Anchore, the Anchore administrator may manage user permissions by managing the roles associated with the user’s identity in Anchore itself. For more information, see [Anchore Enterprise SSO Support](https://docs.anchore.com/current/docs/overview/sso/). See below for an example of the values to provide to Anchore Enterprise for SSO setup: + ```yaml addons: anchore: @@ -108,6 +112,7 @@ Anchore relies on a PostgreSQL database as its primary data store. By default, A Since Anchore relies on a PostgreSQL database, it is recommended that production users utilize their database service's HA and scaling capabilities (e.g. Amazon Aurora, Google Cloud SQL, etc.). For users who need scaling or redundancy for their object storage, S3 or Swift are recommended. By default, Anchore Enterprise will utilize an HA redis deployment, but it can also be configured to use an external redis such as Elasticache). Anchore Enterprise can run one or more analyzer services to scale out the processing of images. There may be many of these analyzers but best practice is to not have more than one per node since analysis is very IO intensive (see [Affinity](https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/anchore-enterprise/-/blob/main/docs/Affinity.md) for an example of how to specify nodeSelector/affinity/anti-affinity). By specifying a `replicaCount` for the analyzers, the number of analysis pods can be scaled up or down: + ```yaml addons: anchore: @@ -115,11 +120,12 @@ addons: anchoreAnalyzer: replicaCount: 2 ``` + As of the anchore-engine Chart version 0.9.0, all services can be scaled-out, so the above example can be modified for the `anchoreApi:`, `anchoreCatalog:`, `anchorePolicyEngine:`, and `anchoreSimpleQueue:` components as well. ### UI -Anchore Enterprise includes a UI, which can be used to scan repositories and images, edit policy bundles, manage users accounts and roles via RBAC, and view and generate security vulnerability and policy evaluation reports. For more information, see [Using the Anchore Enterprirse UI](https://docs.anchore.com/current/docs/using/ui_usage/). +Anchore Enterprise includes a UI, which can be used to scan repositories and images, edit policy bundles, manage users accounts and roles via RBAC, and view and generate security vulnerability and policy evaluation reports. For more information, see [Using the Anchore Enterprise UI](https://docs.anchore.com/current/docs/using/ui_usage/). ### Logging @@ -133,15 +139,17 @@ Anchore Engine and Enterprise expose prometheus metrics in the API of each servi The Big Bang Anchore Helm chart has been modified to use your `monitoring:` values in Big Bang to automatically toggle metrics on/off. -### Healthchecks +### Health Checks Liveness and readiness probes are included in the Anchore Helm chart for all deployments. System health can also be retrieved via the CLI, API, or UI. For example, to see the health of the Anchore services after a Helm install via the CLI: -``` + +```shell kubectl -n anchore exec -it <ANCHORE_ENGINE_API_POD> -- anchore-cli --u <USERNAME> --p <PASSWORD> system status ``` + For more information, see [Anchore Enterprise System Health](https://docs.anchore.com/current/docs/using/ui_usage/system_health/). -### Dependant Packages +### Dependent Packages - PostgreSQL 9.6+ (in-cluster by default; can be configured to use an external postgres) -- Redis (in-cluster by default; can be configured to use an external redis) \ No newline at end of file +- Redis (in-cluster by default; can be configured to use an external redis) diff --git a/charter/packages/authservice/Architecture.md b/charter/packages/authservice/Architecture.md index 6234d3bc6f39b216e1e209985edc38d810a3422f..0bf9cd5d6bf1faaf65afd87eaa04de4fec3b0c57 100644 --- a/charter/packages/authservice/Architecture.md +++ b/charter/packages/authservice/Architecture.md @@ -41,7 +41,7 @@ graph LR end ``` -## Big Bang Touchpoints +## Big Bang Touch Points ### Licensing @@ -49,7 +49,7 @@ graph LR ### Single Sign On -Authservice provides OIDC Single Sign On capabilities for apps that don't have native support. +Authservice provides OIDC Single Sign On capabilities for apps that don't have native support. Pods just need to have istio-injection, a single label which by default is `protect=keycloak` applied to the pods, and a corresponding chain to load into authservice. @@ -118,9 +118,10 @@ There is no UI feature for authservice. Within Big Bang, logs are captured by fluentbit and shipped to elastic by default. -### Healthchecks +### Health Checks The authservice Dockerfile includes a [healthcheck](https://repo1.dso.mil/dsop/istio-ecosystem/authservice/-/blob/master/Dockerfile#L23-24) and the authservice Helm Chart includes [liveness & readiness probes](https://repo1.dso.mil/platform-one/big-bang/apps/core/authservice/-/blob/main/chart/templates/deployment.yaml#L42-47) in its deployment: + ```yaml livenessProbe: tcpSocket: @@ -130,6 +131,6 @@ readinessProbe: port: 10003 ``` -### Dependant Packages +### Dependent Packages -When setting `replicaCount` above `1`, a redis configuration is required. \ No newline at end of file +When setting `replicaCount` above `1`, a redis configuration is required. diff --git a/charter/packages/cluster-auditor/Architecture.md b/charter/packages/cluster-auditor/Architecture.md index eebdacf961360f55685b9764a4af22b0c14eaf66..2df973b2c7d88270bdadd6d770be9fb426a59baa 100644 --- a/charter/packages/cluster-auditor/Architecture.md +++ b/charter/packages/cluster-auditor/Architecture.md @@ -4,7 +4,7 @@ Cluster Auditor(CA) pulls data from the kubernetes API, transforms them and inserts them into Elasticsearch which can then be queried by Kibana. The types of objects are both OPA Gatekeeper CRDs and native kubernetes [objects](https://repo1.dso.mil/platform-one/big-bang/apps/core/cluster-auditor/-/blob/main/chart/templates/configMap.yaml). -## Big Bang Touchpoints +## Big Bang Touch Points ```mermaid graph TB diff --git a/charter/packages/elasticsearch-kibana/Architecture.md b/charter/packages/elasticsearch-kibana/Architecture.md index eacd1224dbbcb02041091555bab381e1ff733aaa..336ce446adeb0e7d6885cb54ef9fac1d5708a2b2 100644 --- a/charter/packages/elasticsearch-kibana/Architecture.md +++ b/charter/packages/elasticsearch-kibana/Architecture.md @@ -2,9 +2,9 @@ ## Overview -[Elasticsearch-Kibana](https://www.elastic.co/elastic-stack) Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Kibana is a data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data. +[Elasticsearch-Kibana](https://www.elastic.co/elastic-stack) Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multi-tenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Kibana is a data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data. -## Big Bang Touchpoints +## Big Bang Touch Points ```mermaid graph TB @@ -117,7 +117,7 @@ logging: ## Licensing -Features like SSO integration, email/slack/pagerduty alerting, FIPS 140-2 mode, encryption at rest, and more for the eck stack requires a platinum or enterprise license. Information about licensing and all features is available [here](https://www.elastic.co/pricing/). A Trial license can be enabled by setting `trial: true` in the below settings to enable a 30-day trial of enterprise settings. +Features like SSO integration, email/slack/Pagerduty alerting, FIPS 140-2 mode, encryption at rest, and more for the eck stack requires a platinum or enterprise license. Information about licensing and all features is available [here](https://www.elastic.co/pricing/). A Trial license can be enabled by setting `trial: true` in the below settings to enable a 30-day trial of enterprise settings. Licensing can be configured with the following values: ```yaml @@ -130,11 +130,11 @@ logging: ## Health Checks -Licensed ECK comes with [built in Health monitoring for Kibana and Elasticsearch](https://www.elastic.co/guide/en/kibana/current/monitoring-kibana.html). This is called self-monitoring within the Kibana UI available at the Stack Monitoring settings https://KIBANA_URL/app/monitoring#. +Licensed ECK comes with [built in Health monitoring for Kibana and Elasticsearch](https://www.elastic.co/guide/en/kibana/current/monitoring-kibana.html). This is called self-monitoring within the Kibana UI available at the Stack Monitoring settings <https://KIBANA_URL/app/monitoring>#. Outside of the UI it is possible to check the health of Elasticsearch cluster via port-forward via doing the following: -```bash +```shell kubectl get secrets -n logging logging-ek-es-elastic-user -o go-template='{{.data.elastic | base64decode}}' kubectl port-forward svc/logging-ek-es-http -n logging 9200:9200 diff --git a/charter/packages/fluentbit/Architecture.md b/charter/packages/fluentbit/Architecture.md index 6d89f630369526954705dff466e4510d34ab5787..c5f5143d3a7c6e3ea334c1cb6c63d3ec17fb3bd7 100644 --- a/charter/packages/fluentbit/Architecture.md +++ b/charter/packages/fluentbit/Architecture.md @@ -4,7 +4,7 @@ [FluentBit](https://fluentbit.io/) is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources, enrich them with filters and send them to multiple destinations. It's the preferred choice for containerized environments like Kubernetes. -## Big Bang Touchpoints +## Big Bang Touch Points ```mermaid graph TB diff --git a/charter/packages/gitlab/Architecture.md b/charter/packages/gitlab/Architecture.md index a78249adadbf4c2de50fe4501286f112023a98b4..e7e174524938e3a3de4708b4c9034eacec940095 100644 --- a/charter/packages/gitlab/Architecture.md +++ b/charter/packages/gitlab/Architecture.md @@ -2,13 +2,13 @@ ## Overview -[Gitlab](https://about.gitlab.com/) is an open-source with premium offering, self-hostable Git respository, build system and container registry. +[Gitlab](https://about.gitlab.com/) is an open-source with premium offering, self-hostable Git repository, build system and container registry. Big Bang's implementation uses the [Gitlab Helm Chart](https://docs.gitlab.com/charts/) to provide custom resources and manage the application. -A more detail view of Bigbang's implementation of Gitlab can be found in the [package docs](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab/-/tree/main/chart/doc). +A more detail view of Big Bang's implementation of Gitlab can be found in the [package docs](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab/-/tree/main/chart/doc). -## Big Bang Touchpoints +## Big Bang Touch Points ### UI @@ -35,34 +35,36 @@ monitoring: Gitlab provides built in health checks. -```bash +```shell GET /-/health ``` Example request -```bash +```shell curl "https://gitlab.example.com/-/health" ``` Gitlab also provides a separate liveness and readiness probes. -```bash +```shell GET /-/readiness GET /-/readiness?all=1 ``` Example request -```bash + +```shell curl "https://gitlab.example.com/-/readiness" ``` -```bash +```shell GET /-/liveness ``` Example request -```bash + +```shell curl "https://gitlab.example.com/-/liveness" ``` @@ -89,13 +91,13 @@ Bigbang currently used the community edition. This can be overwritten in the val edition: ce ``` -More information about Gitlab's licensing can be found [here](https://about.gitlab.com/install/ce-or-ee/) for the information page and [here](https://gitlab.com/gitlab-org/gitlab/blob/master/LICENSE) for the actual license. +More information about GitLab licensing can be found [here](https://about.gitlab.com/install/ce-or-ee/) for the information page and [here](https://gitlab.com/gitlab-org/gitlab/blob/master/LICENSE) for the actual license. ## Storage ### Database Storage -Gitlab uses a Postgresql database to store all metadata for git repositories as well as all business logic around the UI and workflows within the application. By default Bigbang will install a internal Postgres instance to support Gitlab. The reccommended approach is to provision and use an external Postgres instance. +Gitlab uses a Postgresql database to store all metadata for git repositories as well as all business logic around the UI and workflows within the application. By default Bigbang will install a internal Postgres instance to support Gitlab. The recommended approach is to provision and use an external Postgres instance. You can configure an external database by providing the values needed in the Bigbang values.yaml file under the Gitlab section. Entering connection info will automatically disable the deployment of an internal database and will deploy using the external instance. @@ -122,6 +124,7 @@ You can configure an external database by providing the values needed in the Big ### File Storage Gitlab uses S3, Minio, or another S3-style storage for file storage. By default Big Bang deploys an in-cluster Minio instance for this purpose, but you have the option to point to an external Minio or S3 if desired. See the below example for the values to supply: + ```yaml objectStorage: # -- Type of object storage to use for Gitlab, setting to s3 will assume an external, pre-existing object storage is to be used. @@ -150,11 +153,9 @@ Gitlab uses S3, Minio, or another S3-style storage for file storage. By default ## Dependencies -Additional pass throughs for dependencies that deviate from rationalized standards can be passed using the values: tag in the main Bigbang values.yaml. +Additional pass-throughs for dependencies that deviate from rationalized standards can be passed using the values: tag in the main Bigbang values.yaml. ```yaml # -- Values to passthrough to the gitlab runner chart: https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab-runner.git values: {} ``` - - diff --git a/charter/packages/jaeger/Architecture.md b/charter/packages/jaeger/Architecture.md index de62a9e7c8d7f3a2cccc892f73305f7b5ed32282..75223a72afe4c4906cc58ffd6b59c0f6b3cdaa26 100644 --- a/charter/packages/jaeger/Architecture.md +++ b/charter/packages/jaeger/Architecture.md @@ -4,7 +4,7 @@ [Jaeger](https://www.jaegertracing.io/) is an open source implementation of Zipkin that can be used to collect and visualize traces. -## Big Bang Touchpoints +## Big Bang Touch Points ```mermaid graph TB @@ -33,7 +33,7 @@ graph TB ### Storage -When Jaeger recieves traces, it needs a location to store them. The default configuration in the Helm Chart is to use in memory storage. This, of course, doesn't provide High Availability. To provide storage, the chart uses the deployed Elasticserach instance deployed in the logging namespace. +When Jaeger receives traces, it needs a location to store them. The default configuration in the Helm Chart is to use in memory storage. This, of course, doesn't provide High Availability. To provide storage, the chart uses the deployed Elasticsearch instance deployed in the logging namespace. ### Istio Configuration @@ -51,7 +51,7 @@ Istio is configured with knowledge of the jaeger ingest service so istio sidecar ## High Availability -Jaeger is deployed with HorizonalPodAutoscalers for the collector and the queerying pods. Use the below yaml to update the `maxReplicas` on the HPA: +Jaeger is deployed with HorizonalPodAutoscalers for the collector and the querying pods. Use the below yaml to update the `maxReplicas` on the HPA: ```yaml jaeger: @@ -64,10 +64,9 @@ jaeger: maxReplicas: 5 ``` - ## Single Sign on (SSO) -Jaeger does not have built in SSO. In order to provide SSO, this deployment legerages [Authservice](). +Jaeger does not have built in SSO. In order to provide SSO, this deployment leverages [Authservice](https://github.com/istio-ecosystem/authservice). ```mermaid flowchart LR @@ -89,7 +88,7 @@ ingress --> IP subgraph "jaeger namespace" subgraph "jaeger pod" - J["jager"] + J["jaeger"] IP["istio proxy"] --> A IP --> J end @@ -101,10 +100,8 @@ end Jaeger has no licencing options nor requirements. -## Storage - For production workloads, Jaeger uses Elasticsearch to store and query for traces. ## Dependencies -Jaeger can be run without dependencies, but to ensure resilliency of data, it uses Elasticsearch for its span and trace database. +Jaeger can be run without dependencies, but to ensure resiliency of data, it uses Elasticsearch for its span and trace database. diff --git a/charter/packages/mattermost/Architecture.md b/charter/packages/mattermost/Architecture.md index 170fb0de1fc20ef5ebf780b822278094c08242d3..f793d5867316b063ca6edcc92fc97c6fc31edbdc 100644 --- a/charter/packages/mattermost/Architecture.md +++ b/charter/packages/mattermost/Architecture.md @@ -7,6 +7,7 @@ Big Bang's implementation uses the [Mattermost operator](https://github.com/mattermost/mattermost-operator) to provide custom resources and manage the application. ### Basic Tier + ```mermaid graph LR subgraph "Mattermost" @@ -33,6 +34,7 @@ graph LR ``` ### Enterprise Tier with Integrations + ```mermaid graph LR subgraph "Mattermost" @@ -64,11 +66,11 @@ graph LR end ``` -## Big Bang Touchpoints +## Big Bang Touch Points ### UI -The Mattermost UI is the primary way of interacting with Mattermost. The UI is accessible via a web browser, desktop client, and mobile apps. The UI provides access to all mattermost features as well as configuration of the instance via the settings (or "System Console"). +The Mattermost UI is the primary way of interacting with Mattermost. The UI is accessible via a web browser, desktop client, and mobile apps. The UI provides access to all mattermost features as well as configuration of the instance via the settings (or "System Console"). ### Logging @@ -90,19 +92,22 @@ The Mattermost Operator ships by default with health checks on the address `/api **Important Note:** Mattermost by default handles scaling and what it interprets as your needs based upon the users it is configured for. See this note from the Mattermost Operator: - Size defines the size of the Mattermost. This is typically - specified in number of users. This will override replica and resource - requests/limits appropriately for the provided number of users. - values of resources. Accepted values are: 100users, 1000users, 5000users, - 10000users, and 250000users. If replicas and resource requests/limits - are not specified, and Size is not provided the configuration for - 5000users will be applied. Setting ''Replicas'', ''Scheduling.Resources'', - ''FileStore.Replicas'', ''FileStore.Resource'', ''Database.Replicas'', - or ''Database.Resources'' will override the values set by Size. - Setting new Size will override previous values regardless if set - by Size or manually. +```plaintext +Size defines the size of the Mattermost. This is typically +specified in number of users. This will override replica and resource +requests/limits appropriately for the provided number of users. +values of resources. Accepted values are: 100users, 1000users, 5000users, +10000users, and 250000users. If replicas and resource requests/limits +are not specified, and Size is not provided the configuration for +5000users will be applied. Setting ''Replicas'', ''Scheduling.Resources'', +''FileStore.Replicas'', ''FileStore.Resource'', ''Database.Replicas'', +or ''Database.Resources'' will override the values set by Size. +Setting new Size will override previous values regardless if set +by Size or manually. +``` To update the "size" (`users` value) for Mattermost, you need to override the default of 100 in your values (note you do not need to include the word `users` since Big Bang handles this for you), as an example to set 1000: + ```yaml addons: mattermost: @@ -111,6 +116,7 @@ addons: ``` To override Mattermost's handling of replicas and explicitly set replicas you can specify this workaround in your values: + ```yaml addons: mattermost: @@ -126,6 +132,7 @@ SSO is built in for Mattermost and Big Bang uses the [Gitlab SSO integration](ht If using Big Bang's SSO implementation, Keycloak is used behind the scenes to "spoof" the way Gitlab interaction works for SSO. The set up for how to configure Keycloak to handle this is well documented via the [Mattermost docs](https://repo1.dso.mil/platform-one/big-bang/apps/collaboration-tools/mattermost/-/blob/main/docs/keycloak.md). See below for an example of the values to provide to Mattermost for SSO setup: + ```yaml addons: mattermost: @@ -143,6 +150,7 @@ addons: Big Bang deploys the free version of Mattermost by default, but there are two additional tiers of paid licenses for additional features. Pricing for these licenses is typically based upon number of users. Full details can be viewed on [Mattermost's tier page](https://docs.mattermost.com/overview/product.html). If you want to trial the E20 features you can request a trial via Mattermost's [request page](https://mattermost.com/trial/) or after deploying via the System Console you can begin a 30 day trial under the "Edition and License" page. ### Mattermost E10 Additional Features + - Active Directory/LDAP Single Sign-on - OAuth 2.0 authentication for team creation, account creation, and user sign-in - Encrypted push notifications with service level agreements (SLAs) via HPNS @@ -151,6 +159,7 @@ Big Bang deploys the free version of Mattermost by default, but there are two ad - Scale to handle hundreds of users per team ### Mattermost E20 Additional Features + - Advanced SAML 2.0 authentication with Okta, OneLogin, and Active Directory Federation Services - Active Directory/LDAP group sync - OpenID Connect authentication for team creation, account creation, and user sign-in @@ -165,6 +174,7 @@ Big Bang deploys the free version of Mattermost by default, but there are two ad ### License Values Once you have obtained a license this can be added to your values in Big Bang to automatically set up your Mattermost instance with the license (replacing the `license:` value with your full license string): + ```yaml addons: mattermost: @@ -177,7 +187,8 @@ addons: ### Database Storage -Mattermost makes use of a database to store all chat information as well as persistent configuration for all of Mattermost. By default Big Bang deploys an in-cluster Postgresql instance for this purpose, but it is reccomended to point to an external DB instance for HA. Currently Big Bang supports pointing to any external Postgres instance via values config. See the below example for values to point your database connection to an external instance: +Mattermost makes use of a database to store all chat information as well as persistent configuration for all of Mattermost. By default Big Bang deploys an in-cluster Postgresql instance for this purpose, but it is recommended to point to an external DB instance for HA. Currently Big Bang supports pointing to any external Postgres instance via values config. See the below example for values to point your database connection to an external instance: + ```yaml addons: mattermost: @@ -194,6 +205,7 @@ addons: ### File Storage Mattermost uses S3, Minio, or another S3-style storage for file storage. By default Big Bang deploys an in-cluster Minio instance for this purpose, but you have the option to point to an external Minio or S3 if desired. See the below example for the values to supply: + ```yaml addons: mattermost: diff --git a/charter/packages/opa-gatekeeper/Architecture.md b/charter/packages/opa-gatekeeper/Architecture.md index c048bef69ec98b8090b344785aa92ef438b48f0b..d3b03961e09c476e5538fb3e2c8e7036806e3d68 100644 --- a/charter/packages/opa-gatekeeper/Architecture.md +++ b/charter/packages/opa-gatekeeper/Architecture.md @@ -4,7 +4,7 @@ Gatekeeper is an auditing tool that allows administrators to see what resources are currently violating any given policy. -## Big Bang Touchpoints +## Big Bang Touch Points ```mermaid graph LR diff --git a/charter/packages/ref-package/Architecture.md b/charter/packages/ref-package/Architecture.md index 1fe0b5c52d6723027cea645b76c526ce5afbf7ad..0dcf18cadb0f9beac433b748d179866f0f193faa 100644 --- a/charter/packages/ref-package/Architecture.md +++ b/charter/packages/ref-package/Architecture.md @@ -1,9 +1,8 @@ # Big Bang Package Architecture Review -- Big Bang to Package touchpoints / interractions - does the package have a GUI, storage needs, logging, monitoring, health checks, etc +- Big Bang to Package touch points / interactions - does the package have a GUI, storage needs, logging, monitoring, health checks, etc - HA - What is required for HA - SSO - does the package have SSO, is it a licensed feature, if not - is there a strategy to provide rudimentary SSO capability vis AuthService? -- Licensing - describe the licensing model and any tiers of capability that are impacted -- Storage - describe any package specific storage or database requirements -- Dependant packages - list any included dependant packages that will not be elevated to a BB Addon - +- Licensing - describe the licensing model and any tiers of capability that are impacted +- Storage - describe any package specific storage or database requirements +- Dependent packages - list any included dependent packages that will not be elevated to a BB Addon diff --git a/charter/packages/sonarqube/Architecture.md b/charter/packages/sonarqube/Architecture.md index 3c35a6ab97b519f8a1f98000d1f6730b6efec06a..1984b9f19573cc801a043730db7fb4c959635d6a 100644 --- a/charter/packages/sonarqube/Architecture.md +++ b/charter/packages/sonarqube/Architecture.md @@ -4,7 +4,7 @@ [Sonarqube](https://www.sonarqube.org/) is an open-source platform for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs, code smells, and security vulnerabilities. -## Big Bang Touchpoints +## Big Bang Touch Points ```mermaid graph TB @@ -27,7 +27,7 @@ graph TB ### Storage -Persistant storage can be enabled by setting the following values in the bigbang chart: +Persistent storage can be enabled by setting the following values in the bigbang chart: ```yaml addons: @@ -129,4 +129,4 @@ Sonarqube is released under the [Lesser GNU General Public License](https://en.w ## Dependencies Node kernel mods: -https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/master/docs/d_prerequisites.md#sonarqube +<https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/master/docs/d_prerequisites.md#sonarqube> diff --git a/charter/packages/twistlock/Architecture.md b/charter/packages/twistlock/Architecture.md index 072b06c4ca64ed0c3eeb84d0b4c074210981e775..a243a07f1140f56ffe16963aab3d942b22c06e82 100644 --- a/charter/packages/twistlock/Architecture.md +++ b/charter/packages/twistlock/Architecture.md @@ -8,7 +8,7 @@ [Developer Guide](docs/developer-guide.md) -## Big Bang Touchpoints +## Big Bang Touch Points ```mermaid graph LR @@ -29,6 +29,7 @@ graph LR end ``` + ### UI Twistlock Console serves as the user interface within Twistlock. The graphical @@ -38,7 +39,7 @@ user interface (GUI) lets you define policy, configure and control your Twistloc In Bigbang the twistlock defender is installed manual. Follow the document to install defender as a daemonset. -https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/twistlock/-/blob/main/README.md +<https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/twistlock/-/blob/main/README.md> ### Storage @@ -53,6 +54,7 @@ console: ``` ### Database + N/A ### Istio Configuration @@ -91,8 +93,7 @@ SSO can be configured for twistlock manually using the documentation provided. ## Licensing Twistlock deployment requires license to operate. Enter your license key in the twistlock console. \ -[TwistLock License Documentation](https://docs.paloaltonetworks.com/prisma/prisma-cloud/20-04/prisma-cloud-compute-edition-admin/welcome/licensing.html) - +[TwistLock License Documentation](https://docs.paloaltonetworks.com/prisma/prisma-cloud/20-04/prisma-cloud-compute-edition-admin/welcome/licensing.html) ### Health Checks