UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects
Commit 12ef2f09 authored by Caitlin Bowman-Clare's avatar Caitlin Bowman-Clare
Browse files

Update docs/prerequisites/kubernetes-preconfiguration.md

parent d0abeae0
No related branches found
No related tags found
1 merge request!4650Update docs/prerequisites/kubernetes-preconfiguration.md
Hide whitespace changes
Inline Side-by-side
docs/prerequisites/kubernetes-preconfiguration.md
+ 23
− 21
View file @ 12ef2f09
......@@ -2,9 +2,10 @@
## Best Practices
* A CNI (Container Network Interface) that supports Network Policies (which are basically firewalls for the Inner Cluster Network.) (Note: k3d, which is recommended for the quickstart demo, defaults to flannel, which does not support network policies.)
* A Container Network Interface (CNI) that supports Network Policies, which are basically firewalls for the Inner Cluster Network.
**NOTE:** k3d, which is recommended for the quickstart demo, defaults to flannel, which does not support network policies.
* All Kubernetes Nodes and the LB associated with the kube-apiserver should all use private IPs.
* In most case User Application Facing LBs should have Private IP Addresses and be paired with a defense in depth Ingress Protection mechanism like [P1's CNAP](https://p1.dso.mil/#/products/cnap/), a CNAP equivalent (Advanced Edge Firewall), VPN, VDI, port forwarding through a bastion, or air gap deployment.
* In most case User Application Facing LBs should have Private IP Addresses and be paired with a defense in depth Ingress Protection mechanism like [P1's CNAP](https://p1.dso.mil/#/products/cnap/), a CNAP equivalent (e.g., Advanced Edge Firewall), VPN, VDI, port forwarding through a bastion, or air gap deployment.
* CoreDNS in the kube-system namespace should be HA with pod anti-affinity rules
* Master Nodes should be HA and tainted.
* Consider using a licensed Kubernetes Distribution with a support contract.
......@@ -12,26 +13,27 @@
## Service of Type Load Balancer
BigBang's default configuration assumes the cluster you're deploying to supports dynamic load balancer provisioning. Specifically Istio defaults to creating a Kubernetes Service of type Load Balancer, which usually creates an endpoint exposed outside of the cluster that can direct traffic inside the cluster to the istio ingress gateway.
Big Bang's default configuration assumes the cluster you're deploying to supports dynamic load balancer provisioning. Specifically, Istio defaults to creating a Kubernetes Service of type Load Balancer, which usually creates an endpoint exposed outside of the cluster that can direct traffic inside the cluster to the istio ingress gateway.
How Kubernetes service of type LB works depends on implementation details, there are many ways of getting it to work, common methods are listed below:
How Kubernetes service of type LB works depends on implementation details, there are many ways of getting it to work, common methods are listed in the following:
* CSP API Method: (Recommended option for Cloud Deployments)
The Kubernetes Control Plane has a --cloud-provider flag that can be set to aws, azure, etc. If the Kubernetes Master Nodes have that flag set and CSP IAM rights. The control plane will auto provision and configure CSP LBs. (Note: a Vendors Kubernetes Distribution automation, may have IaC/CaC defaults that allow this to work turn key, but if you have issues when provisioning LBs, consult with the Vendor's support for the recommended way of configuring automatic LB provisioning.)
* External LB Method: (Good for bare metal and 0 IAM rights scenarios)
You can override bigbang's helm values so istio will provision a service of type NodePort instead of type LoadBalancer. Instead of randomly generating from the port range of 30000 - 32768, the NodePorts can be pinned to convention based port numbers like 30080 & 30443. If you're in a restricted cloud env or bare metal you can ask someone to provision a CSP LB where LB:443 would map to Nodeport:30443 (of every worker node), etc.
* No LB, Network Routing Methods: (Good options for bare metal)
* CSP API Method (Recommended option for Cloud Deployments):
The Kubernetes Control Plane has a --cloud-provider flag that can be set to aws and/or azure. If the Kubernetes Master Nodes have that flag set and CSP IAM rights. The control plane will auto provision and configure CSP LBs.
**NOTE:** A Vendors Kubernetes Distribution automation, may have IaC/CaC defaults that allow this to work turn key, but if you have issues when provisioning LBs, consult with the Vendor's support for the recommended way of configuring automatic LB provisioning.
* External LB Method (Good for bare metal and 0 IAM rights scenarios):
You can override bigbang's helm values so istio will provision a service of type NodePort instead of type LoadBalancer. Instead of randomly generating from the port range of 30000 - 32768, the NodePorts can be pinned to convention based port numbers like 30080 & 30443. If you're in a restricted cloud env or bare metal, you can ask someone to provision a CSP LB where LB:443 would map to Nodeport:30443 (of every worker node).
* No LB, Network Routing Methods (Good options for bare metal):
* [MetalLB](https://metallb.universe.tf/)
* [kubevip](https://kube-vip.io/)
* [kube-router](https://www.kube-router.io)
## BigBang Doesn’t Support PSPs (Pod Security Policies)
## Big Bang Doesn’t Support Pod Security Policies (PSPs)
* [PSP's are being removed from Kubernetes and will be gone by version 1.25.x](https://repo1.dso.mil/big-bang/bigbang/-/issues/10)
* [PSPs are being removed from Kubernetes and will be gone by version 1.25.x](https://repo1.dso.mil/big-bang/bigbang/-/issues/10)
* [Open Policy Agent Gatekeeper can enforce the same security controls as PSPs](https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/pod-security-policy#pod-security-policies), and is core component of BigBang, which operates as an elevated [validating admission controller](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) to audit and enforce various [constraints](https://github.com/open-policy-agent/frameworks/tree/master/constraint) on all requests sent to the kubernetes api server.
* We recommend users disable PSPs completely given they're being removed, we have a replacement, and PSPs can prevent OPA from deploying (and if OPA is not able to deploy, nothing else gets deployed).
* We recommend users disable PSPs completely given they're being removed, we have a replacement, and PSPs can prevent OPA from deploying. If OPA is not able to deploy, nothing else gets deployed.
* Different ways of Disabling PSPs:
* Edit the kube-apiserver's flags (methods for doing this vary per distribution.)
* Edit the kube-apiserver's flags; methods for doing this vary per distribution.
* ```shell
kubectl patch psp system-unrestricted-psp -p '{"metadata": {"annotations":{"seccomp.security.alpha.kubernetes.io/allowedProfileNames": "*"}}}'
......@@ -42,19 +44,19 @@ You can override bigbang's helm values so istio will provision a service of type
## Kubernetes Distribution Specific Notes
* Note: P1 has forks of various [Kubernetes Distribution Vendor Repos](https://repo1.dso.mil/platform-one/distros), there's nothing special about the P1 forks.
* **NOTE:** P1 has forks of various [Kubernetes Distribution Vendor Repos](https://repo1.dso.mil/platform-one/distros), there's nothing special about the P1 forks.
* We recommend you leverage the Vendors upstream docs in addition to any docs found in P1 Repos; in fact, the Vendor's upstream docs are far more likely to be up to date.
### Kubernetes Version
It is important to note that while Big Bang does not require/mandate usage of a specific Kubernetes Version, we also do not do extensive testing on every version. The below is our general stance on Kubernetes versions:
* Big Bang supports any non-EOL Kubernetes version listed under https://kubernetes.io/releases/
It is important to note that while Big Bang does not require/mandate usage of a specific Kubernetes Version, we also do not do extensive testing on every version. Our general stance on Kubernetes versions is provided in the following:
* Big Bang supports any non-EOL Kubernetes version listed under https://kubernetes.io/releases/.
* Big Bang release and CI testing will primarily be done on the `n-1` minor Kubernetes version (i.e. if 1.27.x is latest, we will test on 1.26.x). We will generally keep our testing environments on the latest patch for that minor version.
* New features added by Kubernetes will be kept behind feature gates until all non-EOL versions support those features
* New features added by Kubernetes will be kept behind feature gates until all non-EOL versions support those features.
### VMWare Tanzu Kubernetes Grid
[Prerequisites section of VMware Kubernetes Distribution Docs's](https://repo1.dso.mil/platform-one/distros/vmware/tkg#prerequisites)
[Prerequisites section of VMware Kubernetes Distribution Docs](https://repo1.dso.mil/platform-one/distros/vmware/tkg#prerequisites)
### Cluster API
......@@ -84,7 +86,7 @@ It is important to note that while Big Bang does not require/mandate usage of a
kubectl get daemonset istio-cni-node -n kube-system -o json | jq '.spec.template.spec.containers[] += {"securityContext":{"privileged":true}}' | kubectl replace -f -
```
1. Modify the OpenShift cluster(s) with the following scripts based on <https://istio.io/v1.7/docs/setup/platform-setup/openshift/>
1. Modify the OpenShift cluster(s) with the following scripts based on <https://istio.io/v1.7/docs/setup/platform-setup/openshift/>.
```shell
# Istio Openshift configurations Post Install
......@@ -116,7 +118,7 @@ It is important to note that while Big Bang does not require/mandate usage of a
* RKE2 turns PSPs on by default (see above for tips on disabling)
* RKE2 sets selinux to enforcing by default ([see os-preconfiguration.md for selinux config](os-preconfiguration.md))
Since BigBang makes several assumptions about volume and load balancing provisioning by default, it's vital that the rke2 cluster must be properly configured. The easiest way to do this is through the in tree cloud providers, which can be configured through the `rke2` configuration file such as:
Since BigBang makes several assumptions about volume and load balancing provisioning by default, it's vital that the rke2 cluster must be properly configured. The easiest way to do this is through the in tree cloud providers, which can be configured through the `rke2` configuration file such as:
```yaml
# aws, azure, gcp, etc...
......@@ -128,4 +130,4 @@ cloud-provider-config: ...
For example, if using the aws terraform modules provided [on repo1](https://repo1.dso.mil/platform-one/distros/rancher-federal/rke2/rke2-aws-terraform), setting the variable: `enable_ccm = true` will ensure all the necessary resources tags.
In the absence of an in-tree cloud provider (such as on-prem), the requirements can be met by ensuring a default storage class and automatic load balancer provisioning exist.
In the absence of an in-tree cloud provider (e.g., on-prem), the requirements can be met by ensuring a default storage class and automatic load balancer provisioning exist.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

UNCLASSIFIED - NO CUI