UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects
Commit 5c02e64b authored by Caitlin Bowman-Clare's avatar Caitlin Bowman-Clare Committed by Michael Martin
Browse files

Update docs/developer/package-integration/flux.md,...

parent 752fd797
No related branches found
No related tags found
1 merge request!4558Update docs/developer/package-integration/flux.md,...
# Istio Hardened
Big Bang has added the `.Values.istio.hardened` map attibute to the values of applications that can be istio-injected (when `.Values.istio.enabled` is `true`). This document walks through the impact of setting `.Values.istio.hardened: true` on how traffic is managed within a given istio-injected package.
Big Bang has added the `.Values.istio.hardened` map attibute to the values of applications that can be istio-injected (when `.Values.istio.enabled` is `true`). This document walks through the impact of setting `.Values.istio.hardened: true` on how traffic is managed within a given istio-injected package.
## Prerequisites
In order for `.Values.istio.hardened.enabled: true` to have any impact, the package must also have `.Values.istio.enabled: true` set. This is because all of the resources created by setting `.Values.istio.hardened.enabled: true` are applied to the istio service mesh, which includes istio sidecar proxies. If there are no istio proxies, then no mesh components exist in the namespace and therefore istio Kubernetes resources in the namespace will not effect anything.
In order for `.Values.istio.hardened.enabled: true` to have any impact, the package must also have `.Values.istio.enabled: true` set. This is because all of the resources created by setting `.Values.istio.hardened.enabled: true` are applied to the istio service mesh, which includes istio sidecar proxies. If there are no istio proxies, then no mesh components exist in the namespace and therefore istio Kubernetes resources in the namespace will not effect anything.
## REGISTRY_ONLY Istio Sidecar resources
When `.Values.istio.hardened.enabled: true` is set, a `Sidecar` resource is applied to the package's namespace that sets the outboundTrafficPolicy of the Sidecar to `REGISTRY_ONLY`. What this means is that for pods with an istio-proxy running as a "sidecar", the only egress traffic allowed is for traffic that is destinated for a service that exists within the istio service mesh registry.
When `.Values.istio.hardened.enabled: true` is set, a `Sidecar` resource is applied to the package's namespace that sets the outboundTrafficPolicy of the Sidecar to `REGISTRY_ONLY`. What this means is that for pods with an istio-proxy running as a "sidecar", the only egress traffic allowed is for traffic that is destinated for a service that exists within the istio service mesh registry.
By default, all Kubernetes Services are added to this registry. However, cluster-external hostnames, IP addresses, and other endpoints will NOT be reachable with this Sidecar in place. For example, if an application attempts to reach out to the Kubernetes API Service at `kubernetes.default.svc.cluster.local` (or any of it's SANs), the request will not be blocked by the Sidecar. Conversely, if the application attempts to reach out to s3.us-gov-west-1.amazonaws.com, the request with fail unless there is a ServiceEntry (see below) that adds s3.us-gov-west-1.amazonaws.com to the service mesh registry. This Sidecar is added in order to provide defense in depth, working alongside NetworkPolicies to prevent data exfiltration by malicious actors.
By default, all Kubernetes Services are added to this registry. However, cluster-external hostnames, IP addresses, and other endpoints will NOT be reachable with this Sidecar in place. For example, if an application attempts to reach out to the Kubernetes API Service at `kubernetes.default.svc.cluster.local` (or any of it's SANs), the request will not be blocked by the Sidecar. Conversely, if the application attempts to reach out to s3.us-gov-west-1.amazonaws.com, the request with fail unless there is a ServiceEntry (see below) that adds s3.us-gov-west-1.amazonaws.com to the service mesh registry. This Sidecar is added in order to provide defense in depth, working alongside NetworkPolicies to prevent data exfiltration by malicious actors.
## ServiceEntry Istio resources
Because some application have well-documented requirements to reach out to cluster external endpoints (S3 is one common example), Big Bang has added ServiceEntries to get those endpoints included in the Istio service registry. If we missed one, please open an issue detailing what endpoint needs to be whitelisted with a ServiceEntry. Alternatively, you can create your own whitelisted endpoints by using the `.Values.istio.hardened.customServiceEntries` list, which will generate a ServiceEntry according to the `.spec` map you set.
Because some application have well-documented requirements to reach out to cluster external endpoints (S3 is one common example), Big Bang has added ServiceEntries to get those endpoints included in the Istio service registry. If we missed one, please open an issue detailing what endpoint needs to be whitelisted with a ServiceEntry. Alternatively, you can create your own whitelisted endpoints by using the `.Values.istio.hardened.customServiceEntries` list, which will generate a ServiceEntry according to the `.spec` map you set.
> `customServiceEntries` is there for *edge cases* that may be specific to your requirements, and not all `customServiceEntries` may be appropriate for all Big Bang users.
......@@ -56,7 +56,7 @@ spec:
For more information on writting ServiceEntries, see [this documentation](https://istio.io/latest/docs/reference/config/networking/service-entry/)
## Authorization Policies
[Istio Authorization Policies](https://istio.io/latest/docs/reference/config/security/authorization-policy/#AuthorizationPolicy) will be created provided `istio.enabled` and `istio.hardened.enabled` are set to `true`. There is a default deny policy which will deny everything that is not explicitly allowed with another policy. Denials look like a 403 with the message `RBAC: access denied`. Other policies that are created might include allow ingress gateways, allow monitoring, or allow a supported service that needs access to these resources. You will find these listed under `istio.hardened` as named objects that have 3 properties: `enabled`, `namespaces`, and `principals`. There are also templates which allow you to create custom authorization policies through additional values, these are described in greater detail below. The last rules to note are global rules. These are any rules created in the `istio-system` namespace. Rather than affecting just the `istio-system` namespace, they will apply to all namespaces.
[Istio Authorization Policies](https://istio.io/latest/docs/reference/config/security/authorization-policy/#AuthorizationPolicy) will be created provided `istio.enabled` and `istio.hardened.enabled` are set to `true`. There is a default deny policy which will deny everything that is not explicitly allowed with another policy. Denials look like a 403 with the message `RBAC: access denied`. Other policies that are created might include allow ingress gateways, allow monitoring, or allow a supported service that needs access to these resources. You will find these listed under `istio.hardened` as named objects that have three properties: `enabled`, `namespaces`, and `principals`. There are also templates which allow you to create custom authorization policies through additional values, these are described in greater detail below. The last rules to note are global rules. These are any rules created in the `istio-system` namespace. Rather than affecting just the `istio-system` namespace, they will apply to all namespaces.
### Rules
Apart from the default deny, most rules will be explicit allows. Included rules will be for other supported packages. Any other rules will need to be created with the templates described below. Rules affect a namespace. Rules go on the "server" in the "client-server" relationship.
......@@ -108,4 +108,4 @@ spec:
- source:
namespaces:
- "my-namespace"
```
\ No newline at end of file
```
......@@ -4,21 +4,21 @@ Following the steps in this guide will result in the `integration` job being run
If there's a need to disable this job, add `SKIP INTEGRATION` to the title of the Merge Request.
Big Bang uses a continuous deployment tool, [Flux](https://fluxcd.io/) to deploy packages using Helm charts sourced from Git ([GitOps](https://www.weave.works/technologies/gitops/)). This document will cover how to integrate a Helm chart, from a mission application or other package, into the Flux pattern required by Big Bang. Once complete, you will be able to deploy your package with Big Bang.
Big Bang uses a continuous deployment tool, [Flux](https://fluxcd.io/) to deploy packages using Helm charts sourced from Git ([GitOps](https://www.weave.works/technologies/gitops/)). This document will cover how to integrate a Helm chart, from a mission application or other package, into the Flux pattern required by Big Bang. Once complete, you will be able to deploy your package with Big Bang.
## Prerequisites
- [Helm](https://helm.sh/docs/intro/install/)
- [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
- A multi-node Kubernetes cluster to deploy Big Bang and your package
- A [Big Bang project containing the upstream Helm chart](./upstream.md)
* [Helm](https://helm.sh/docs/intro/install/)
* [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
* [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
* A multi-node Kubernetes cluster to deploy Big Bang and your package
* A [Big Bang project containing the upstream Helm chart](./upstream.md)
> Throughout this document, we will be setting up an application called `podinfo` as a demonstration
> Throughout this document, we will be setting up an application called `podinfo` as a demonstration.
## Big Bang Helm Chart
The purpose of the Big Bang Helm chart is to create a Big Bang compatible, easy-to-use spec for deploying the package. Reasonable and safe defaults are provided and any needed secrets are auto-created. We accept the trade-off of easy deployment for complicated template code. Details are in the following steps.
The purpose of the Big Bang Helm chart is to create a Big Bang compatible, easy-to-use spec for deploying the package. Reasonable and safe defaults are provided and any needed secrets are auto-created. We accept the trade-off of easy deployment for complicated template code. Details are provided in the following:
```shell
gitrepository.yaml # Flux GitRepository, configured by Big Bang chart values.
......@@ -59,7 +59,7 @@ EOF
### Namespace
The package will be deployed in its own namespace. BigBang pre-creates this namespace so that labels and annotations can be controlled. Setup `bigbang/templates/$PKGNAME/namespace.yaml` with the following:
The package will be deployed in its own namespace. Big Bang pre-creates this namespace so that labels and annotations can be controlled. Set up `bigbang/templates/$PKGNAME/namespace.yaml` with the following:
```yaml
{{- $pkg := "podinfo" }}
......@@ -86,7 +86,7 @@ podinfo:
#### GitRepository
Flux's source controller uses the [GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) resource to pull Helm chart changes from Git. Use the [GitRepository API Specification](https://fluxcd.io/docs/components/source/gitrepositories/#specification) to create a `GitRepository` resource named `bigbang/templates/$PKGNAME/gitrepository.yaml` with the following content:
Flux's source controller uses the [GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) resource to pull Helm chart changes from Git. Use the [GitRepository API Specification](https://fluxcd.io/docs/components/source/gitrepositories/#specification) to create a `GitRepository` resource named `bigbang/templates/$PKGNAME/gitrepository.yaml` with the following content:
```yaml
{{- $pkg := "podinfo" }}
......@@ -129,7 +129,7 @@ podinfo:
#### HelmRelease
Big Bang exclusively uses Helm charts for deployment through Flux. Using the [HelmRelease API Specification](https://fluxcd.io/docs/components/helm/helmreleases/#specification), create a `HelmRelease` resource named `bigbang/templates/$PKGNAME/helmrelease.yaml` with the following content:
Big Bang exclusively uses Helm charts for deployment through Flux. Using the [HelmRelease API Specification](https://fluxcd.io/docs/components/helm/helmreleases/#specification), create a `HelmRelease` resource named `bigbang/templates/$PKGNAME/helmrelease.yaml` with the following content:
```yaml
{{- $pkg := "podinfo" }}
......@@ -186,7 +186,7 @@ podinfo:
### ImagePullSecret
Big Bang images are pulled from Iron Bank. In order to provide credentials for Iron Bank, Big Bang will create a secret for each package called `private-registry`. In `bigbang/templates/$PKGNAME/imagepullsecret.yaml`, add the following content:
Big Bang images are pulled from Iron Bank. In order to provide credentials for Iron Bank, Big Bang will create a secret for each package called `private-registry.` In `bigbang/templates/$PKGNAME/imagepullsecret.yaml,` add the following content:
```yaml
{{- $pkg := "podinfo" }}
......@@ -207,22 +207,22 @@ data:
{{- end }}
```
> Other secrets can be added for credentials, certificates, etc. by creating a file named `bigbang/templates/$PKGNAME/secret-<name>.yaml`. Big Bang is responsible for creating these secrets using values from the user. More details are included in the integration documentation for databases, object stores, sso, etc.
> Other secrets can be added for credentials and/or certificates by creating a file named `bigbang/templates/$PKGNAME/secret-<name>.yaml`. Big Bang is responsible for creating these secrets using values from the user. More details are included in the integration documentation for databases, object stores, and/or SSO.
### Package Values
Package values (chart/values.yaml) should contain upstream values plus any placeholders of values needed for Big Bang.The following guidelines should be used when adding values to the package:
- Assume the package will be run without Big Bang. Values enabling features from other packages (e.g. metrics, ingress, SSO) should be turned off by default. Big Bang will enable them through overrides.
- Re-use existing chart values rather than adding new ones when possible.
- Only change the default values from the upstream Helm chart when necessary.
- Comment any changes made to the upstream Helm values so it is clear that the changes should carry forward on upgrades.
- Assume that Big Bang will create secrets (e.g. TLS certificates, credentials) and provide the reference to the chart.
- Create blank placeholders for Big Bang values to avoid Helm errors during deployment.
* Assume the package will be run without Big Bang. Values enabling features from other packages (e.g., metrics, ingress, and/or SSO) should be turned off by default. Big Bang will enable them through overrides.
* Re-use existing chart values rather than adding new ones when possible.
* Only change the default values from the upstream Helm chart when necessary.
* Comment any changes made to the upstream Helm values so it is clear that the changes should carry forward on upgrades.
* Assume that Big Bang will create secrets (e.g., TLS certificates and/or credentials) and provide the reference to the chart.
* Create blank placeholders for Big Bang values to avoid Helm errors during deployment.
### Override Values
Big Bang has a few options for overwriting values in packages. The package's `HelmRelease`, that we created earlier, contains a `ValuesFrom` section that references a secret with `common`, `default`, and `overlay` keys. Each of these keys can contain a set of override values that get passed down to the package. Here is a table explaining the difference between the possible overlays:
Big Bang has a few options for overwriting values in packages. The package's `HelmRelease`, that we created earlier, contains a `ValuesFrom` section that references a secret with `common`, `default`, and `overlay` keys. Each of these keys can contain a set of override values that get passed down to the package. Here is a table explaining the difference between the possible overlays:
| Name | Description | Source | Priority |
| --------- | ----------------------------------------------- | -------------------------------------------- | --------- |
......@@ -261,7 +261,7 @@ podinfo:
flux: {}
```
Use the Big Bang default values to make sure our Helm templates don't have any syntax errors. Run the following:
Use the Big Bang default values to make sure our Helm templates don't have any syntax errors. Run the following:
```shell
# Get the helm chart
......@@ -274,7 +274,7 @@ Use the Big Bang default values to make sure our Helm templates don't have any s
### Validation
To validate that the Helm chart is working, perform the following steps to deploy your package. This assumes you already have a Kubernetes cluster running.
To validate that the Helm chart is working, perform the following steps to deploy your package. This assumes you already have a Kubernetes cluster running.
1. Disable all packages that are enabled by default in Big Bang by adding the following to `bigbang/values.yaml`
......@@ -318,7 +318,7 @@ To validate that the Helm chart is working, perform the following steps to deplo
1. To enable Big Bang packages that are disabled by default, add the appropriate code block for the Big Bang package from the Big Bang helm chart [values file](https://repo1.dso.mil/big-bang/bigbang/-/blob/master/chart/values.yaml).
- For example, if you want to test a package's functionality with MinIO, you would add this block from the Big Bang helm chart values file to the package repo's `bigbang/values.yaml` file
* For example, if you want to test a package's functionality with MinIO, you would add this block from the Big Bang helm chart values file to the package repo's `bigbang/values.yaml` file.
```yaml
addons:
......@@ -329,14 +329,14 @@ To validate that the Helm chart is working, perform the following steps to deplo
enabled: true
```
- This would deploy MinIO in the `integration` stage with default configurations
* This would deploy MinIO in the `integration` stage with default configurations
- Any values that are present in the Big Bang helm chart values file can be configured here also. The pipeline merges these two files into a single values file before deployment, so the package repo's `bigbang/values.yaml` provides a way to configure a Third Party or Sandbox package along with Big Bang package configurations in a single values file.
* Any values that are present in the Big Bang helm chart values file can be configured here also. The pipeline merges these two files into a single values file before deployment, so the package repo's `bigbang/values.yaml` provides a way to configure a Third Party or Sandbox package along with Big Bang package configurations in a single values file.
- A `tests/test-values.yaml` in a package repo can be used as override values for the pipeline. It allows pipeline-specific configurations so that the package's `chart/values.yaml` doesn't have to be changed.
* A `tests/test-values.yaml` in a package repo can be used as override values for the pipeline. It allows pipeline-specific configurations so that the package's `chart/values.yaml` doesn't have to be changed.
1. Install flux using the [instructions from Big Bang](https://repo1.dso.mil/big-bang/bigbang/-/blob/master/docs/guides/deployment-scenarios/quickstart.md#step-8-install-flux).
1. Install the package using the bigbang Helm chart
1. Install the package using the bigbang Helm chart.
```shell
helm upgrade -i -n bigbang --create-namespace -f ~/bigbang/chart/values.yaml -f bigbang/values.yaml bigbang-podinfo bigbang
......@@ -348,7 +348,7 @@ To validate that the Helm chart is working, perform the following steps to deplo
watch kubectl get gitrepo,hr,po -A
```
1. Troubleshoot any errors
1. Troubleshoot any errors.
```shell
kubectl get events -A
......@@ -356,7 +356,7 @@ To validate that the Helm chart is working, perform the following steps to deplo
> If you are using a private Git repository or pulling images from a private image repository, you will need to add credentials into the `git.credentials.username`/`git.credentials.password` and/or `registryCredentials.username`/`registryCredentials.password` using the `--set` option for Helm.
1. Cleanup cluster
1. Clean up cluster.
```shell
helm delete -n bigbang bigbang-podinfo
......@@ -380,9 +380,9 @@ To validate that the Helm chart is working, perform the following steps to deplo
### Installation
1. Install Big Bang
1. Install Big Bang.
`helm upgrade -i -n bigbang --create-namespace -f ~/bigbang/chart/values.yaml -f bigbang/values.yaml bigbang ~/bigbang/chart`
1. Install this chart
2. Install this chart.
`helm upgrade -i -n bigbang --create-namespace -f ~/bigbang/chart/values.yaml -f bigbang/values.yaml bigbang-podinfo bigbang`
### Removal
......@@ -391,7 +391,7 @@ To validate that the Helm chart is working, perform the following steps to deplo
```
1. Commit your changes
1. Commit your changes.
> If you are developing something different than `podinfo`, run `grep -ir podinfo` to make sure your replaced all of the instances with your application name.
......
# Helm Package Standards
This document describes the technical guidelines that should be in place when building a Helm chart and integrating it with BigBang.
This document describes the technical guidelines that should be in place when building a Helm chart and integrating it with Big Bang.
## Helm Package Versioning Scheme
Big Bang packages follow a standard semantic versioning scheme for both the package tag and the chart version. The package tag will always be in line with the chart version (not the `appVersion`). To distinguish between BigBang specific changes within the semantic version of the upstream chart, a suffix of `-bb.#` will be added to _all_ charts and tags.
Big Bang packages follow a standard semantic versioning scheme for both the package tag and the chart version. The package tag will always be in line with the chart version (not the `appVersion`). To distinguish between BigBang specific changes within the semantic version of the upstream chart, a suffix of `-bb.#` will be added to _all_ charts and tags.
For example, for the upstream [`istio-operator`](https://github.com/istio/istio/tree/1.7.3/manifests/charts/istio-operator) pinned at version `1.7.3`, the big bang version (with the modified `values.yaml` for an Iron Bank image) will be tagged `1.7.3-bb.0`. If in the same `istio-operator` release, BigBang requires chart modifications (such as to support `imagePullSecrets`), then the new version becomes `1.7.3-bb.1`.
For example, for the upstream [`istio-operator`](https://github.com/istio/istio/tree/1.7.3/manifests/charts/istio-operator) pinned at version `1.7.3`, the Big Bang version (with the modified `values.yaml` for an Iron Bank image) will be tagged `1.7.3-bb.0`. If in the same `istio-operator` release, Big Bang requires chart modifications (such as to support `imagePullSecrets`), then the new version becomes `1.7.3-bb.1`.
For another example in using the [`kube-prometheus-stack`](https://github.com/prometheus-community/helm-charts/tree/kube-prometheus-stack-12.2.2/charts/kube-prometheus-stack), the upstream is versioned at `12.2.2`, meaning BigBang's initial fork will be `12.2.2-bb.0`. Future additions, such as adding `VirtualServices` for the ingresses, bumps to the `-bb.#` will happen in sequence every time BigBang updates the chart within the same version.
For another example in using the [`kube-prometheus-stack`](https://github.com/prometheus-community/helm-charts/tree/kube-prometheus-stack-12.2.2/charts/kube-prometheus-stack), the upstream is versioned at `12.2.2`, meaning BigBang's initial fork will be `12.2.2-bb.0`. Future additions, such as adding `VirtualServices` for the ingresses, bumps to the `-bb.#` will happen in sequence every time BigBang updates the chart within the same version.
## Big Bang Values File
* In the values.yaml file [here](../../../chart/values.yaml), each package should have its own region at `.package_name` if its in Core or `.addons.package_name`.
* In the values.yaml file [here](../../../chart/values.yaml), each package should have its own region at `.package_name` if its in Core or `.addons.package_name.`
* User Interface:
* If there exists need for ingress traffic into the package, the package should create a VirtualService conditional on the existence of `istio.enabled` being set to true. This value should default to false. The BigBang chart should set this true for all packages
* There should be a region under the package for configuring SSO that looks like this when there are multiple packages
* If there exists need for ingress traffic into the package, the package should create a VirtualService conditional on the existence of `istio.enabled` being set to true. This value should default to false. The BigBang chart should set this true for all packages.
* There should be a region under the package for configuring SSO that looks like this when there are multiple packages.
```yaml
sso:
......@@ -37,10 +37,10 @@ For another example in using the [`kube-prometheus-stack`](https://github.com/pr
client_secret: "change_me"
```
* If sso is enabled and a value is not provided in the SSO configuration of the package, it should default to the top level SSO configuration
* If sso is enabled and a value is not provided in the SSO configuration of the package, it should default to the top level SSO configuration.
* Database Connections:
* The BigBang chart should prevent the use of a database bundled as part of the package chart by default, and warn if an end user uses one anyways.
* There should be a database section under the package configuration that matches the following section
* There should be a database section under the package configuration that matches the following section.
```yaml
database:
......@@ -53,14 +53,14 @@ For another example in using the [`kube-prometheus-stack`](https://github.com/pr
type: "" # Optional. One of mysql, mssql, postgres, mongo if ther
```
* Monitoring
* Charts should expect a value `monitoring.enabled` to be set by the BigBang chart to conditionally create monitoring components (`ServiceMonitors`, `PodMonitors`, etc). This value should default to false
* Monitoring:
* Charts should expect a value `monitoring.enabled` to be set by the BigBang chart to conditionally create monitoring components (e.g., `ServiceMonitors` and/or `PodMonitors`). This value should default to false.
## Secrets
* The BigBang chart should make an `ImagePullSecret` in the namespace the package will be deployed in.
* If the package chart cannot accept credentials (e.g. for databases) as a value, then the BigBang chart should make the secret with values passed into BigBang and pass the Secret to the package chart by name.
* If the package chart cannot accept credentials (e.g., for databases) as a value, then the BigBang chart should make the secret with values passed into BigBang and pass the Secret to the package chart by name.
## Big Bang Helm Release
......@@ -97,9 +97,9 @@ commonLabels:
## Big Bang Package Readme Generation
Follow [this guide](https://repo1.dso.mil/big-bang/product/packages/gluon/-/blob/master/docs/bb-package-readme.md?ref_type=heads) for package readme.md generation
Follow [this guide](https://repo1.dso.mil/big-bang/product/packages/gluon/-/blob/master/docs/bb-package-readme.md?ref_type=heads) for package readme.md generation.
Note the Big Bang package README.md is separate from the README.md included as part of the upstream chart. See ArgoCD for an example, [Big Bang package README.md](https://repo1.dso.mil/big-bang/product/packages/argocd/-/blob/main/README.md?ref_type=heads) vs [upstream chart README.md](https://repo1.dso.mil/big-bang/product/packages/argocd/-/blob/main/chart/README.md?ref_type=heads)
Note the Big Bang package README.md is separate from the README.md included as part of the upstream chart. See ArgoCD for an example, [Big Bang package README.md](https://repo1.dso.mil/big-bang/product/packages/argocd/-/blob/main/README.md?ref_type=heads) vs [upstream chart README.md](https://repo1.dso.mil/big-bang/product/packages/argocd/-/blob/main/chart/README.md?ref_type=heads).
Each package value in values.yaml should have a comment descriptor above the value. We generate the package README.md using a script that expects this format. The README.md will contain a table with default configurations and descriptors pulled from the comments.
......@@ -111,7 +111,7 @@ strategy: scalable
## Kubernetes Objects
These requirements for the kubernetes components come from the Kubernetes STIG, Kubesec.io and other best practices
These requirements for the kubernetes components come from the Kubernetes STIG, Kubesec.io and other best practices.
* Resource Limits and Requests set for cpu and memory and they are [Guaranteed QoS](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed)
* Containers are not run in privileged mode
......
# Monitoring
Monitoring packages requires a way to scrape metrics, provide those to data storage, and analyzing the results. Big Bang uses Prometheus and Grafana as the service for monitoring. Most packages offer built-in Prometheus metrics scraping or an add-on that will scrape the metrics. This document will show you how to integrate metrics scraping with Big Bang.
Monitoring packages requires a way to scrape metrics, provide those to data storage, and analyze the results. Big Bang uses Prometheus and Grafana as the service for monitoring. Most packages offer built-in Prometheus metrics scraping or an add-on that will scrape the metrics. This document will show you how to integrate metrics scraping with Big Bang.
## Prerequisites
Before integrating with Prometheus, you must identify the following:
- Does the application support metrics exporting for Prometheus. If not, you will need to find a Prometheus exporter to provide this service.
- Does the upstream Helm chart for the application (or exporter) support Prometheus natively? If not, we'll have to create our own monitoring resources.
> Searching the Helm chart for `monitoring.coreos.com` will usually find any resources that support Prometheus
- What path and port are used to scrape metrics on the application or exporter?
- What services and/or pods are deployed that should be monitored?
- Is there a pre-existing Grafana dashboard that can be leveraged? If not, we will need to create one.
* Does the application support metrics exporting for Prometheus? If not, you will need to find a Prometheus exporter to provide this service.
* Does the upstream Helm chart for the application (or exporter) support Prometheus natively? If not, we'll have to create our own monitoring resources.
> Searching the Helm chart for `monitoring.coreos.com` will usually find any resources that support Prometheus.
* What path and port are used to scrape metrics on the application or exporter?
* What services and/or pods are deployed that should be monitored?
* Is there a pre-existing Grafana dashboard that can be leveraged? If not, we will need to create one.
## Integration
### Placeholder values
The package requires placeholder values for whether the monitoring stack (e.g. Prometheus / Grafana) is enabled and what label to use for dashboards. In `chart/values.yaml`, add placeholders for these:
The package requires placeholder values for whether the monitoring stack (e.g., Prometheus/Grafana) is enabled and what label to use for dashboards. In `chart/values.yaml`, add placeholders for these:
```yaml
serviceMonitor:
......@@ -30,11 +30,11 @@ serviceMonitor:
label: grafana_dashboard
```
> In this case, we put the values under `serviceMonitor:` since it already exists in the upstream Helm chart. Otherwise, we would create `monitoring:` for the values
> In this case, we put the values under `serviceMonitor:` since it already exists in the upstream Helm chart. Otherwise, we would create `monitoring:` for the values.
### Pass down values
Big Bang needs to set the placeholders above to the appropriate values. In addition, upstream charts may already have values related to monitoring that need to be set.
Big Bang needs to set the placeholders above to the appropriate values. In addition, upstream charts may already have values related to monitoring that need to be set.
In `bigbang/templates/podinfo/values.yaml`, add the following to pass down the values from Big Bang to PodInfo.
......@@ -48,7 +48,7 @@ serviceMonitor:
### Dependency
If we plan to scrape metrics from the application with the monitoring stack, we need to make sure the monitoring stack is deployed first so that CRDs are in place before we deploy our resources. To do this, we add a `dependsOn` section in the `bigbang/templates/podinfo/helmrelease.yaml` file like this:
If we plan to scrape metrics from the application with the monitoring stack, we need to make sure the monitoring stack is deployed first so that CRDs are in place before we deploy our resources. To do this, we add a `dependsOn` section in the `bigbang/templates/podinfo/helmrelease.yaml` file like this:
```yaml
spec:
......@@ -69,7 +69,7 @@ spec:
### Service Monitor
If the upstream Helm chart provides you with a `ServiceMonitor` and `Service` for scraping metrics, verify that there is a conditional around each one to only deploy them if monitoring is enabled (e.g. `{{- if .Values.serviceMonitor.enabled }}` or `{{- if .Values.monitoring.enabled }}`)
If the upstream Helm chart provides you with a `ServiceMonitor` and `Service` for scraping metrics, verify that there is a conditional around each one to only deploy them if monitoring is enabled (e.g., `{{- if .Values.serviceMonitor.enabled }}` or `{{- if .Values.monitoring.enabled }}`).
If the upstream chart does **not** provide a `ServiceMonitor` and `Service` for scraping metrics, you will need to create one yourself using the [Prometheus instructions for running an exporter](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/running-exporters.md).
......@@ -81,20 +81,20 @@ If the application is using Role Based Access Control (RBAC), you may need to cr
### Alerts
Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service. By creating a `PrometheusRule`, you can configure these conditions for your application.
Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service. By creating a `PrometheusRule`, you can configure these conditions for your application.
You will need to decide what aspects of the application should be monitored and alerted on to detect potential failures in the service it provides. Some examples include:
You will need to decide what aspects of the application should be monitored and alerted on to detect potential failures in the service it provides. Several examples are listed in the following:
- Low disk space on a persistent volume
- Loss of connectivity to external resources
- Metrics cannot be scraped
- Operator down
- Pods in CrashLookBackOff state
- Pods restarting too often
- Latency too high
- Web application returns 4xx or 5xx too often
- No log messages for too long
- Pod memory too close to limit
* Low disk space on a persistent volume
* Loss of connectivity to external resources
* Metrics cannot be scraped
* Operator down
* Pods in CrashLookBackOff state
* Pods restarting too often
* Latency too high
* Web application returns 4xx or 5xx too often
* No log messages for too long
* Pod memory too close to limit
All of these rules must be based on [PromQL queries](https://prometheus.io/docs/prometheus/latest/querying/basics/) using the application's metrics.
......@@ -106,7 +106,7 @@ Some examples of rules can be found in the [Big Bang monitoring chart](https://r
Dashboards are important for administrators to understand what is happening in your package and when action needs to be taken.
1. Create a dashboard
1. Create a dashboard:
Some packages or maintainers provide Grafana dashboards upstream, otherwise you can search [Grafana's Dashboard Repository](https://grafana.com/grafana/dashboards/) for a relevant Dashboard. If there is already a ready-made Grafana dashboard for your package provided upstream, you should use [Kpt](https://googlecontainertools.github.io/kpt/installation/) to sync it into monitoring package (for example flux provides the JSON dashboards in their upstream repo):
......@@ -115,9 +115,9 @@ Dashboards are important for administrators to understand what is happening in y
kpt pkg get https://github.com/fluxcd/flux2.git//manifests/monitoring/grafana/dashboards@v0.9.1 chart/dashboards/
```
If you need to create your own dashboard, open Grafana and use `Create > Dashboard`. Add a panel and setup the query to pull custom data from your package or general data about your pods (e.g. container_processes). After you have saved your dashboard in Grafana, use `Share (icon) > Export` to save the dashboard to a .json file in `chart/dashboards`. You can leave the `Export for sharing externally` slider off.
If you need to create your own dashboard, open Grafana and use `Create > Dashboard`. Add a panel and setup the query to pull custom data from your package or general data about your pods (e.g., container_processes). After you have saved your dashboard in Grafana, use `Share (icon) > Export` to save the dashboard to a .json file in `chart/dashboards`. You can leave the `Export for sharing externally` slider off.
1. We will store dashboards in a ConfigMap for Grafana's sidecar to parse. Create a ConfigMapList in `chart/templates/bigbang/dashboards.yaml` to store all of the dashboards:
1. We will store dashboards in a ConfigMap for Grafana's sidecar to parse. Create a ConfigMapList in `chart/templates/bigbang/dashboards.yaml` to store all of the dashboards:
```yaml
{{- $pkg := "podinfo" }}
......@@ -145,7 +145,7 @@ Dashboards are important for administrators to understand what is happening in y
{{- end }}
```
> Podinfo's Helm chart already had a key for monitoring named `serviceMonitor`. You may need to use a different key or create one named `monitoring`.
> Podinfo's Helm chart already had a key for monitoring named `serviceMonitor.` You may need to use a different key or create one named `monitoring.`
1. Commit your dashboard files:
......@@ -155,7 +155,7 @@ Dashboards are important for administrators to understand what is happening in y
git push
```
1. If your package is being integrated as a supported application in BigBang, you can add your Dashboards to the core monitoring package.
1. If your package is being integrated as a supported application in Big Bang, you can add your Dashboards to the core monitoring package.
Create a new folder within `chart/dashboards/APP_NAME` and sync your JSON files for your dashboard(s) there, whether using KPT from a Github repo or individual files from Grafana's Dashboard Repository.
......@@ -173,7 +173,7 @@ Dashboards are important for administrators to understand what is happening in y
### Setup
Monitoring must be enabled in our Big Bang deployment and our application. We do this by setting `monitoring.enabled`: `true` in `bigbang/values.yaml`. Then, deploy Big Bang and your application to your cluster.
Monitoring must be enabled in our Big Bang deployment and our application. We do this by setting `monitoring.enabled`: `true` in `bigbang/values.yaml`. Then, deploy Big Bang and your application to your cluster.
```shell
# This assumes you have the Big Bang repository cloned in ~/bigbang
......@@ -194,18 +194,18 @@ curl -L https://prometheus.bigbang.dev
curl -L https://grafana.bigbang.dev
```
> If your application also has an ingress, test it (e.g. `https://podinfo.bigbang.dev`)
> If your application also has an ingress, test it (e.g. `https://podinfo.bigbang.dev`).
### Target
Open `https://prometheus.bigbang.dev` and navigate to `Status > Targets`. The `State` should show `UP` if metrics are being scraped for your package.
Open `https://prometheus.bigbang.dev` and navigate to `Status > Targets.` The `State` should show `UP` if metrics are being scraped for your package.
> There should be one `Endpoint` for every replica pod of your package.
### Alert Rules
In Prometheus, navigate to `Alerts`. Verify that the `PrometheusRule` alerting rules show up here and are green.
In Prometheus, navigate to `Alerts.` Verify that the `PrometheusRule` alerting rules show up here and are green.
### Dashboards
Open `https://grafana.bigbang.dev` and navigate to `Dashboards > Manage`. Make sure your dashboards are listed. Select each one and verify that it is working correctly.
Open `https://grafana.bigbang.dev` and navigate to `Dashboards > Manage.` Make sure your dashboards are listed. Select each one and verify that it is working correctly.
# Network Policies
To increase the overall security posture of Big Bang, network policies are put in place to only allow ingress and egress from package namespaces to other needed services. A deny by default policy is put in place to deny all traffic that is not explicitly allowed. The following is how to implement the network policies per Big Bang standards.
To increase the overall security posture of Big Bang, network policies are put in place to only allow ingress and egress from package namespaces to other needed services. A deny by default policy is put in place to deny all traffic that is not explicitly allowed. The following is how to implement the network policies per Big Bang standards.
## Table of Contents
......@@ -9,7 +9,7 @@ To increase the overall security posture of Big Bang, network policies are put i
## Prerequisites
- Understanding of ports and communications of applications and other components within BigBang
- `chart/templates/bigbang` and `chart/templates/bigbang/networkpolicies` folders within package for committing bigbang specific templates
- `chart/templates/bigbang` and `chart/templates/bigbang/networkpolicies` folders within package for committing bigbang specific templates.
## Integration
......@@ -65,9 +65,9 @@ spec:
### Was Something Important Blocked?
There are a few ways to determine if a network policy is blocking egress or ingress to or from a pod.
- Test things from the pod's perspective using ssh/exec. See [this portion](../../guides/deployment-scenarios/sso-quickstart.md#step-18-update-inner-cluster-dns-on-the-workload-cluster) of the keycloak quickstart for an example of how do to that.
- Curl a pod's IP from another pod to see if network polices are blocking that traffic. Use `kubectl pod -o wide -n <podNamespace>` to see pod IP addresses.
- Check the pod logs (or curl from one container to the service) for a `context deadline exceeded` or `connection refused` message.
* Test things from the pod's perspective using ssh/exec. See [this portion](../../guides/deployment-scenarios/sso-quickstart.md#step-18-update-inner-cluster-dns-on-the-workload-cluster) of the keycloak quickstart for an example of how do to that.
* Curl a pod's IP from another pod to see if network polices are blocking that traffic. Use `kubectl pod -o wide -n <podNamespace>` to see pod IP addresses.
* Check the pod logs (or curl from one container to the service) for a `context deadline exceeded` or `connection refused` message.
### Allowing Exceptions
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment