Big Bang has added the `.Values.istio.hardened` map attibute to the values of applications that can be istio-injected (when `.Values.istio.enabled` is `true`). This document walks through the impact of setting `.Values.istio.hardened: true` on how traffic is managed within a given istio-injected package.
Big Bang has added the `.Values.istio.hardened` map attibute to the values of applications that can be istio-injected, when `.Values.istio.enabled` is `true`. This document walks through the impact of setting `.Values.istio.hardened: true` on how traffic is managed within a given istio-injected package.
## Prerequisites
In order for `.Values.istio.hardened.enabled: true` to have any impact, the package must also have `.Values.istio.enabled: true` set. This is because all of the resources created by setting `.Values.istio.hardened.enabled: true` are applied to the istio service mesh, which includes istio sidecar proxies. If there are no istio proxies, then no mesh components exist in the namespace and therefore istio Kubernetes resources in the namespace will not effect anything.
In order for `.Values.istio.hardened.enabled: true` to have any impact, the package must also have `.Values.istio.enabled: true` set. This is because all of the resources created by setting `.Values.istio.hardened.enabled: true` are applied to the istio service mesh, which includes istio sidecar proxies. If there are no istio proxies, then no mesh components exist in the namespace and therefore istio Kubernetes resources in the namespace will not effect anything.
## REGISTRY_ONLY Istio Sidecar resources
When `.Values.istio.hardened.enabled: true` is set, a `Sidecar` resource is applied to the package's namespace that sets the outboundTrafficPolicy of the Sidecar to `REGISTRY_ONLY`. What this means is that for pods with an istio-proxy running as a "sidecar", the only egress traffic allowed is for traffic that is destinated for a service that exists within the istio service mesh registry.
## REGISTRY_ONLY Istio Sidecar Resources
When `.Values.istio.hardened.enabled: true` is set, a `Sidecar` resource is applied to the package's namespace that sets the outboundTrafficPolicy of the Sidecar to `REGISTRY_ONLY`. What this means is that for pods with an istio-proxy running as a "sidecar", the only egress traffic allowed is for traffic that is destinated for a service that exists within the istio service mesh registry.
By default, all Kubernetes Services are added to this registry. However, cluster-external hostnames, IP addresses, and other endpoints will NOT be reachable with this Sidecar in place. For example, if an application attempts to reach out to the Kubernetes API Service at `kubernetes.default.svc.cluster.local`(or any of it's SANs), the request will not be blocked by the Sidecar. Conversely, if the application attempts to reach out to s3.us-gov-west-1.amazonaws.com, the request with fail unless there is a ServiceEntry (see below) that adds s3.us-gov-west-1.amazonaws.com to the service mesh registry. This Sidecar is added in order to provide defense in depth, working alongside NetworkPolicies to prevent data exfiltration by malicious actors.
By default, all Kubernetes Services are added to this registry. However, cluster-external hostnames, IP addresses, and other endpoints will NOT be reachable with this Sidecar in place. For example, if an application attempts to reach out to the Kubernetes API Service at `kubernetes.default.svc.cluster.local`, or any of it's SANs, the request will not be blocked by the Sidecar. Conversely, if the application attempts to reach out to s3.us-gov-west-1.amazonaws.com, the request with fail unless there is a ServiceEntry (refer to the example below) that adds s3.us-gov-west-1.amazonaws.com to the service mesh registry. This Sidecar is added in order to provide defense in depth, working alongside NetworkPolicies to prevent data exfiltration by malicious actors.
## ServiceEntry Istio resources
Because some application have well-documented requirements to reach out to cluster external endpoints (S3 is one common example), Big Bang has added ServiceEntries to get those endpoints included in the Istio service registry. If we missed one, please open an issue detailing what endpoint needs to be whitelisted with a ServiceEntry. Alternatively, you can create your own whitelisted endpoints by using the `.Values.istio.hardened.customServiceEntries` list, which will generate a ServiceEntry according to the `.spec` map you set.
## ServiceEntry Istio Resources
Because some application have well-documented requirements to reach out to cluster external endpoints (e.g., S3 is one common example), Big Bang has added ServiceEntries to get those endpoints included in the Istio service registry. If we missed one, please open an issue detailing what endpoint needs to be whitelisted with a ServiceEntry. Alternatively, you can create your own whitelisted endpoints by using the `.Values.istio.hardened.customServiceEntries` list, which will generate a ServiceEntry according to the `.spec` map you set.
> `customServiceEntries` is there for *edge cases* that may be specific to your requirements, and not all `customServiceEntries` may be appropriate for all Big Bang users.
### Example customServiceEntry
To create a ServiceEntry for google, the corresponding customServiceEntry attribute could be set:
### Example CustomServiceEntry
To create a ServiceEntry for google, the corresponding customServiceEntry attribute could be set to the following:
```yaml
istio:
enabled:true
...
...
@@ -56,7 +56,7 @@ spec:
For more information on writting ServiceEntries, see [this documentation](https://istio.io/latest/docs/reference/config/networking/service-entry/)
## Authorization Policies
[Istio Authorization Policies](https://istio.io/latest/docs/reference/config/security/authorization-policy/#AuthorizationPolicy) will be created provided `istio.enabled` and `istio.hardened.enabled` are set to `true`. There is a default deny policy which will deny everything that is not explicitly allowed with another policy. Denials look like a 403 with the message `RBAC: access denied`. Other policies that are created might include allow ingress gateways, allow monitoring, or allow a supported service that needs access to these resources. You will find these listed under `istio.hardened` as named objects that have 3 properties: `enabled`, `namespaces`, and `principals`. There are also templates which allow you to create custom authorization policies through additional values, these are described in greater detail below. The last rules to note are global rules. These are any rules created in the `istio-system` namespace. Rather than affecting just the `istio-system` namespace, they will apply to all namespaces.
[Istio Authorization Policies](https://istio.io/latest/docs/reference/config/security/authorization-policy/#AuthorizationPolicy) will be created provided `istio.enabled` and `istio.hardened.enabled` are set to `true`. There is a default deny policy which will deny everything that is not explicitly allowed with another policy. Denials look like a 403 with the message `RBAC: access denied`. Other policies that are created might include allow ingress gateways, allow monitoring, or allow a supported service that needs access to these resources. You will find these listed under `istio.hardened` as named objects that have three properties: `enabled`, `namespaces`, and `principals`. There are also templates which allow you to create custom authorization policies through additional values, these are described in greater detail below. The last rules to note are global rules. These are any rules created in the `istio-system` namespace. Rather than affecting just the `istio-system` namespace, they will apply to all namespaces.
### Rules
Apart from the default deny, most rules will be explicit allows. Included rules will be for other supported packages. Any other rules will need to be created with the templates described below. Rules affect a namespace. Rules go on the "server" in the "client-server" relationship.
# This doc is to detail the Big Bang integration testing strategy for changes to Big Bang images by the container hardening team
# Big Bang Integration Testing Strategy
## Using Big Bang repository to test changes to Big Bang images
This document is to detail the Big Bang integration testing strategy for changes to Big Bang images by the container hardening team.
Developers can leverage the Big Bang repository CI pipelines to test the integration with Big Bang core components. Big Bang CI installs core Big Bang applications by default plus additional applications using MR labels. Example, adding the Nexus label to a Big Bang MR will install Big Bang core (istio, kyverno, monitoring, tempo, kiali, neuvector, promtail, loki and grafana) plus Nexus Repository.
## Using Big Bang repository to Test Changes to Big Bang images
## To perform an integration test with Big Bang core:
Developers can leverage the Big Bang repository Continuous Integration (CI) pipelines to test the integration with Big Bang core components. Big Bang CI installs core Big Bang applications by default plus additional applications using Merge Request (MR) labels. For example, adding the Nexus label to a Big Bang MR will install Big Bang core (e.g., istio, kyverno, monitoring, tempo, kiali, neuvector, promtail, loki and/or grafana) plus Nexus Repository.
1. Create branch in the [Big Bang repository](https://repo1.dso.mil/big-bang/bigbang)
2. Add an override in [bigbang/chart/values.yaml](../../chart/values.yaml) for the image and repository to be tested on your branch. Example below, if testing Nexus, navigate to the Nexus configuration section and modify the values with image and repository overrides. Our CI can pull images from ironbank or ironbank-staging within registry1.dso.mil.
## Big Bang Core Integration Test
## Find section in Big Bang chart values
To perform an integration test with Big Bang core, complete the following:
1. Create branch in the [Big Bang repository](https://repo1.dso.mil/big-bang/bigbang).
2. Add an override in [bigbang/chart/values.yaml](../../chart/values.yaml) for the image and repository to be tested on your branch. An example is provided below. If testing Nexus, navigate to the Nexus configuration section and modify the values with image and repository overrides. Our CI can pull images from ironbank or ironbank-staging within registry1.dso.mil.
### Find section in Big Bang chart values:
NexusRepositoryManager:
values: {}
## Replace with
### Replace with:
NexusRepositoryManager:
values:
...
...
@@ -26,19 +30,20 @@ Big Bang package values can differ from this example. Navigate to the specific B
3. Push branch to repo1.
4. Create draft MR from your branch into master branch. Be sure to leave MR in draft status and do not include a review label. if testing a package not included in Big Bang core, add the appropriate label for that package. Fill out the description with details such as testing x.y.z image - container hardening team.
4. Create draft Merge Request (MR) from your branch into master branch. Be sure to leave the MR in draft status and do not include a review label. If you're testing a package not included in Big Bang core, add the appropriate label for that package. Fill out the description with details, such as testing x.y.z image - container hardening team.
5. Creating an MR should trigger a CI run under the pipelines tab. If the correct package label was added prior to opening the MR, the pipeline should install that package. Review the results of your MR pipeline. Two pipeline jobs are created in the same stage to test the Big Bang umbrella helm chart installation.
5. Creating an MR should trigger a CI run under the pipelines tab. If the correct package label was added prior to opening the MR, the pipeline should install that package. Review the results of your MR pipeline. 2 pipeline jobs are created in the same stage to test the Big Bang umbrella helm chart installation.
**Clean install: Installs Big Bang core, plus any packages included in labels off of the branch being tested. This tests a clean install of Big Bang to a k3d cluster.**
## Clean install - installs big bang core plus any packages included in labels off of the branch being tested. This tests a clean install of Big Bang to a k3d cluster.
## Upgrade - installs big bang core plus any packages included in labels off of master. Followed by a helm upgrade to the branch being tested. This tests the upgrade path on a k3d cluster.
**Upgrade: Installs Big Bang core, plus any packages included in labels off of master. Followed by a helm upgrade to the branch being tested. This tests the upgrade path on a k3d cluster.**
6. Reviewing results of pipeline -> navigate to the output of the pipeline jobs.
## Things to observe from pipeline output:
### Things to observe from pipeline output:
- Expand the 03_wait_for_helmreleases.sh. At the bottom of this section, users can review that all pods were running on the cluster at completion of this script.
- Collapse the previous section and review the 03_helm_tests.sh. This section should contain results of our CI tests on the live applications installed by Big Bang. Each package testing is different. Packages with frontends typically include a cypress test. Cypress test artifacts can typically be reviewed from the Job Artifacts. Navigate to browse job artifacts to watch a .mp4 video file of the cypress test for your package.
- Some packages do not include cypress tests. Some do not include tests at all (istioOperator etc..) In this case we recommend validating that the package installed to the cluster and that the tenant application ran successfully in the CI job output. Verify pods running using the 03_wait_for_helmreleases.sh section and all helmReleases succeeded using the Helmreleases section.
* Expand the 03_wait_for_helmreleases.sh. At the bottom of this section, users can review that all pods were running on the cluster at completion of this script.
* Collapse the previous section and review the 03_helm_tests.sh. This section should contain results of our CI tests on the live applications installed by Big Bang. Each package testing is different. Packages with frontends typically include a cypress test. Cypress test artifacts can typically be reviewed from the Job Artifacts. Navigate to browse job artifacts to watch a .mp4 video file of the cypress test for your package.
* Some packages do not include cypress tests. Some do not include tests at all (e.g., istioOperator) In this case we recommend validating that the package installed to the cluster and that the tenant application ran successfully in the CI job output. Verify pods running using the 03_wait_for_helmreleases.sh section and all helmReleases succeeded using the Helmreleases section.
7. Once testing is complete, close MR, branch will be cleaned up automatically.
\ No newline at end of file
7. Once testing is complete, close the MR and the branch will be cleaned up automatically.
