diff --git a/docs/guides/using-bigbang/autoscaling-fluentbit-and-promtail.md b/docs/guides/using-bigbang/autoscaling-fluentbit-and-promtail.md index aebfc8f2f2801554fab1aa884f07669dca42a538..cdbb827faab65fb9000897c55a2d1f1a0ebf2ef5 100644 --- a/docs/guides/using-bigbang/autoscaling-fluentbit-and-promtail.md +++ b/docs/guides/using-bigbang/autoscaling-fluentbit-and-promtail.md @@ -1,12 +1,13 @@ -# Installing and Configuring VPA for Vertical Scaling of Fluent Bit and Promtail Pods +# Installing and Configuring Vertical Pod Autoscaler (VPA) for Vertical Scaling of Fluent Bit and Promtail Pods -Since log forwarder pods like fluentbit & promtail are designed to have 1 pod per node via a ReplicaSet instantiation they are unable to be Horizontally Scaled when reaching their resource limits. They can still be Vertically scaled once [VPA](https://repo1.dso.mil/big-bang/product/packages/vpa) is installed. +Since log forwarder pods, like Fluentbit and Promtail, are designed to have one pod per node via a ReplicaSet instantiation, they are unable to be horizontally scaled when reaching their resource limits. They can still be vertically scaled once [VPA](https://repo1.dso.mil/big-bang/product/packages/vpa) is installed. -# 1.Prerequisites: -- Kubernetes cluster running version 1.14 or later -- kubectl command-line tool configured to access the cluster +# 1. Prerequisites -# 2.Install the Vertical Pod Autoscaler +- Kubernetes cluster must be running version 1.14 or later. +- Kubectl command-line tool must be configured to access the cluster. + +# 2. Install the Vertical Pod Autoscaler 1. Run the following commands to install the VPA components from the package repo: @@ -14,7 +15,7 @@ Since log forwarder pods like fluentbit & promtail are designed to have 1 pod pe cd vpa helm install vertical-pod-autoscaler chart/ - 2. Add package via the BigBang 2.0 `packages:` key: + 2. Add package via the Big Bang 2.0 `packages:` key: packages: vpa: @@ -47,6 +48,7 @@ autoscaling: ``` # 4.Configure Promtail Deployment + For Promtail, make sure the following settings are added to the Packages Helm Chart [values.yaml](https://repo1.dso.mil/big-bang/product/packages/promtail/-/blob/main/chart/values.yaml) file: ```yaml @@ -73,6 +75,7 @@ vpa: ``` # 5.Verify VPA Status + Check the VPA status to ensure it is functioning correctly and providing recommendations. Run the following command: @@ -84,7 +87,7 @@ Run the following command: Look for the Conditions section and verify that the status is Healthy. # 6.Monitor and Observe Scaling -Monitor the cluster and observe how the VPA scales the Fluent Bit and Promtail pods based on resource utilization. -Review the pod resource utilization metrics using commands like kubectl top pods or monitoring tools like Prometheus and Grafana. -By following these steps, you should be able to install and configure the Vertical Pod Autoscaler (VPA) to vertically scale Fluent Bit and Promtail pods in your Kubernetes cluster when additional resources are needed and headroom is available. Adjust the configurations and policies based on your specific requirements and application characteristics. +Monitor the cluster and observe how the VPA scales the Fluent Bit and Promtail pods based on resource utilization. Review the pod resource utilization metrics using commands like kubectl top pods or monitoring tools like Prometheus and Grafana. + +By following these steps, you should be able to install and configure the VPA to vertically scale Fluent Bit and Promtail pods in your Kubernetes cluster when additional resources are needed and headroom is available. Adjust the configurations and policies based on your specific requirements and application characteristics. diff --git a/docs/guides/using-bigbang/default-credentials.md b/docs/guides/using-bigbang/default-credentials.md index 79eacf5c26658247971a2e622e35ae940243bb07..c0eae1f7b367654cc377345f9a8289a8dd26b1e1 100644 --- a/docs/guides/using-bigbang/default-credentials.md +++ b/docs/guides/using-bigbang/default-credentials.md @@ -1,8 +1,8 @@ # Credentials for Big Bang Packages -This document includes details on credentials to access each package in a default install (without SSO). It is safe to assume that any packages not listed in the two categories below either have no need for authentication or use different methods (ex: Velero require kubectl access). +This document includes details on credentials to access each package in a default install (i.e., without SSO). It is safe to assume that any packages not listed in the two categories below either have no need for authentication or use different methods (e.g., Velero require kubectl access). -## Packages With No Built in Authentication +## Packages With No Built-in Authentication Although the below applications have no built in authentication, Big Bang's helm values can be configured to deploy authservice in front of these endpoints. Authservice is an Authentication Proxy that can integrate with SSO providers like Keycloak. @@ -10,9 +10,9 @@ Although the below applications have no built in authentication, Big Bang's helm - Monitoring (Prometheus) - Monitoring (Alertmanager) -## Packages With Built in Authentication +## Packages With Built-in Authentication -The applications in the table below provide both SSO and built in auth. The table gives default credentials and ways to access and/or override those. +The applications in the table below provide both SSO and built-in authentication. The table gives default credentials and ways to access and/or override those. | Package (Application) | Default Username | Default Password | Additional Notes | | --------------------- | ---------------- | ---------------- | ---------------- | diff --git a/docs/guides/using-bigbang/efk-plg-logging-migration.md b/docs/guides/using-bigbang/efk-plg-logging-migration.md index e97741806cb6109bd96f498f888565995ef7b62d..783da0d7e549fe87ee6151a7c88c8f8dc497d1ce 100644 --- a/docs/guides/using-bigbang/efk-plg-logging-migration.md +++ b/docs/guides/using-bigbang/efk-plg-logging-migration.md @@ -1,14 +1,13 @@ # Switching Between EFK and PLG -These instructions detail how to switch between EFK and PLG logging solutions. +These instructions detail how to switch between Elasticsearch, Fluentbit, and Kibana (EFK) and Promtail, Loki, and Grafana (PLG) logging solutions. The EFK stack is an open-source choice for the Kubernetes log aggregation and analysis and is comprised of the following: - Elasticsearch is a distributed and scalable search engine commonly used to sift through large volumes of log data. - Fluentbit is a log shipper. It is an open source log collection agent which support multiple data sources and output formats. -- Kibana is a UI tool for querying, data visualization and dashboards. +- Kibana is a User Interface (UI) tool for querying, data visualization and dashboards. -Today the EFK stack (Elasticsearch, Fluentbit, and Kibana) is enabled by default in the bigbang chart. -The EFK stack appears within the chart as follows: +Today the EFK stack is enabled by default in the Big Bang chart. The EFK stack appears within the chart as shown in the following: ```yaml @@ -26,7 +25,8 @@ fluentbit: ``` -If you want to use a logging solution that doesn't require a license and has a smaller footprint, Big Bang provides PLG. PLG is comprised of: +If you want to use a logging solution that doesn't require a license and has a smaller footprint, Big Bang provides PLG. PLG is comprised of the following: + - Promtail is an agent that detects targets (e.g., local log files), attaches labels to log streams from the pods, and ships them to Loki. - Loki is an open-source, multi-tenant log aggregation system. It can be used with Grafana and Promtail to collect and access logs. - Grafana is an open-source visualization platform that processes time-series data from Loki and makes the logs accessible in a web UI. @@ -65,5 +65,4 @@ tempo: ``` -NOTE: -Both Fluentbit and Promtail forward logs to Grafana Loki. Should you want a logging stack comprised solely of Grafana technologies, the PLG stack will accommodate. The use of Fluentbit to send logs (log forwarder) is still a viable option rather than Promtail, and is configured within Big Bang to ship to Loki if enabled. Big Bang's recommendation is to use Fluentbit as a log forwarder because it is more feature rich. Promtail can only send to a limited set of endpoints (e.g., Loki and S3), whereas Fluentbit can send to numerous endpoints. +**NOTE:** Both Fluentbit and Promtail forward logs to Grafana Loki. Should you want a logging stack comprised solely of Grafana technologies, the PLG stack will accommodate. The use of Fluentbit to send logs (i.e., log forwarder) is still a viable option rather than Promtail, and is configured within Big Bang to ship to Loki if enabled. Big Bang's recommendation is to use Fluentbit as a log forwarder because it is more feature rich. Promtail can only send to a limited set of endpoints (e.g., Loki and S3), whereas Fluentbit can send to numerous endpoints. diff --git a/docs/guides/using-bigbang/image-pull-policy.md b/docs/guides/using-bigbang/image-pull-policy.md index d78650c4249a94ad1369571eca1d757fe3b8e500..8d6822526de15a5e396e1096db2d0135c75c1bc9 100644 --- a/docs/guides/using-bigbang/image-pull-policy.md +++ b/docs/guides/using-bigbang/image-pull-policy.md @@ -4,7 +4,7 @@ Big Bang is currently working to standardize the adoption of a global image pull policy so that customers can set a single value and have it passed to all packages. -The global image pull policy has been adopted in Big Bang for the core packages and for addons. In the Big Bang values.yaml file, a global parameter has been created to set the global image pull policy (`imagePullPolicy` in values) and it gets passed down to all core packages and addons spec. The default value for this global policy is `IfNotPresent`. +The global image pull policy has been adopted in Big Bang for the core packages and for addons. In the Big Bang values.yaml file, a global parameter has been created to set the global image pull policy (`imagePullPolicy` in values) and it gets passed down to all core packages and add-ons spec. The default value for this global policy is `IfNotPresent`. We have also documented the package overrides required if you want to set a single package/pod with a different pull policy than the global. diff --git a/docs/guides/using-bigbang/kyverno-use-by-apps.md b/docs/guides/using-bigbang/kyverno-use-by-apps.md index e83179a0d464cb58c050e774273d57db01918323..d50e5129afa621f7cf78d30bc2b2acca23f7d4c6 100644 --- a/docs/guides/using-bigbang/kyverno-use-by-apps.md +++ b/docs/guides/using-bigbang/kyverno-use-by-apps.md @@ -1,13 +1,17 @@ # List Of Big Bang Applications That Are Using Kyverno ## Gitlab Runner -In order for Gitlab Runner auto registration to work the runner must be deployed in the `gitlab` namespace or have a copy of the 'gitlab-runner-secret' in its separate namespace. The Big Bang helm chart deploys Gitlab Runner in a separate namespace. Gitlab Runner uses Kyverno to copy the gitlab runner secret. There are alternate options if Kyverno is not wanted. -1. By default Gitlab Runner uses Kyverno to support auto registration + +In order for Gitlab Runner auto registration to work, the runner must be deployed in the `gitlab` namespace or have a copy of the 'gitlab-runner-secret' in its separate namespace. The Big Bang helm chart deploys Gitlab Runner in a separate namespace. Gitlab Runner uses Kyverno to copy the gitlab runner secret. There are alternate options if Kyverno is not wanted. + +1. By default Gitlab Runner uses Kyverno to support auto registration. 2. The runner token can be added to value overrides. 3. The gitlab-runner-secret can be manually copied to the `gitlab-runner` namespace. ## Fluentbit -In order for Fluentbit to automatically have a connection setup for for Elastic the Elastic root password and certificate secrets must be copied from the `logging` namespace to `fluentbit`. Kyverno is leveraged to copy these automatically. There are alternatives if using Kyverno is not desired: + +In order for Fluentbit to automatically have a connection set up for Elastic, the Elastic root password and certificate secrets must be copied from the `logging` namespace to `fluentbit`. Kyverno is leveraged to copy these automatically. There are alternatives if using Kyverno is not desired. These alternatives are listed in the following: + 1. By default Kyverno is used for auto-connection. 2. `fluentbit.values.additionalOutputs.elasticsearch` could be used to setup a connection with Elastic rather than the auto-connection. 3. Secrets can be manually copied to the `fluentbit` namespace. diff --git a/docs/guides/using-bigbang/monitoring-applications-with-annotations-scraping.md b/docs/guides/using-bigbang/monitoring-applications-with-annotations-scraping.md index 96c2b0f1445455e80d41c569183bfbcdcd2532dc..8a60567964ea04a13dabf50a2b254684dc8922ad 100644 --- a/docs/guides/using-bigbang/monitoring-applications-with-annotations-scraping.md +++ b/docs/guides/using-bigbang/monitoring-applications-with-annotations-scraping.md @@ -1,10 +1,10 @@ -# Integrating Prometheus Label Scraping for sample application +# Integrating Prometheus Label Scraping for Sample Application ## Introduction -Integrating Prometheus metrics scraping with label scraping services is helpful for monitoring monitoring applications that either don't ship with a a ServiceMonitor, for testing in researching to build out a serviceMonitor resource or in-development applications where creating and managing annotations is simpler than building and managing ServiceMonitors. This guide will explain how to integrate metrics scraping for your annotated service endpoints or specific pods with Big Bang. +Integrating Prometheus metrics scraping with label scraping services is helpful for monitoring monitoring applications that either don't ship with a a ServiceMonitor, for testing in researching to build out a serviceMonitor resource, or in-development applications where creating and managing annotations is simpler than building and managing ServiceMonitors. This guide will explain how to integrate metrics scraping for your annotated service endpoints or specific pods with Big Bang. -By default, global endpoint monitoring is behind a hidden value in Big Bang. If you wish to enable it the sections below are for each configuration. +By default, global endpoint monitoring is behind a hidden value in Big Bang. If you wish to enable, it the sections below are for each configuration. The available Global serviceMonitor for annotated endpoints service is defined by the prometheus job name `kubernetes-service-endpoints` with following hidden value `monitoring.globalServiceEndpointMetrics` in `values.yaml`: ``` @@ -33,7 +33,7 @@ Before integrating with Prometheus, ensure the following: ### 1. Define Placeholder Values for a Service (Recommended) Add placeholders in `chart/values.yaml` to configure whether the monitoring stack (Prometheus) is enabled: -the following is an example of a placeholder for allowing scraping of the `metrics-server` service endpoint: +the following is an example of a placeholder for allowing scraping of the `metrics-server` service endpoint: ```yaml addons: metrics-server: @@ -46,10 +46,10 @@ addons: serviceMonitor: enabled: false # set to true to enable monitoring if the service is not already being scraped. This is an easily re-producible example but ideally the app will not ship with a serviceMonitor. -note: The example above is for the metrics-server application, you will need to update the service annotations to match your application. Also note that metrics-server is already being scraped by a local serviceMonitor, so you will need to disable it so label scraping can be enabled. +**NOTE:** The example above is for the metrics-server application, you will need to update the service annotations to match your application. Also note that metrics-server is already being scraped by a local serviceMonitor, so you will need to disable it so label scraping can be enabled. ``` -Applications configured correctly will be scraped by Prometheus under the `target->` `kubernetes-service-endpoints` (see image below) in the Prometheus UI. If you do not know the correct port and path for your application, you may use `Service Discovery` in the Prometheus UI to find the correct port and path for your application. +Applications configured correctly will be scraped by Prometheus under the `target->` `kubernetes-service-endpoints` in the Prometheus User Interface (UI). Refer to the image below. If you do not know the correct port and path for your application, you may use `Service Discovery` in the Prometheus UI to find the correct port and path for your application. - +