UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects
Commit 3da4a060 authored by Michael Martin's avatar Michael Martin
Browse files

Merge branch 'cmbc_10july' into 'master'

Update docs/guides/using-bigbang/autoscaling-fluentbit-and-promtail.md,...

See merge request !4707
parents 3653cac7 ca58dd41
No related branches found
No related tags found
1 merge request!4707Update docs/guides/using-bigbang/autoscaling-fluentbit-and-promtail.md,...
Pipeline #3476443 failed
# Installing and Configuring VPA for Vertical Scaling of Fluent Bit and Promtail Pods
# Installing and Configuring Vertical Pod Autoscaler (VPA) for Vertical Scaling of Fluent Bit and Promtail Pods
Since log forwarder pods like fluentbit & promtail are designed to have 1 pod per node via a ReplicaSet instantiation they are unable to be Horizontally Scaled when reaching their resource limits. They can still be Vertically scaled once [VPA](https://repo1.dso.mil/big-bang/product/packages/vpa) is installed.
Since log forwarder pods, like Fluentbit and Promtail, are designed to have one pod per node via a ReplicaSet instantiation, they are unable to be horizontally scaled when reaching their resource limits. They can still be vertically scaled once [VPA](https://repo1.dso.mil/big-bang/product/packages/vpa) is installed.
# 1.Prerequisites:
- Kubernetes cluster running version 1.14 or later
- kubectl command-line tool configured to access the cluster
# 1. Prerequisites
# 2.Install the Vertical Pod Autoscaler
- Kubernetes cluster must be running version 1.14 or later.
- Kubectl command-line tool must be configured to access the cluster.
# 2. Install the Vertical Pod Autoscaler
1. Run the following commands to install the VPA components from the package repo:
......@@ -14,7 +15,7 @@ Since log forwarder pods like fluentbit & promtail are designed to have 1 pod pe
cd vpa
helm install vertical-pod-autoscaler chart/
2. Add package via the BigBang 2.0 `packages:` key:
2. Add package via the Big Bang 2.0 `packages:` key:
packages:
vpa:
......@@ -47,6 +48,7 @@ autoscaling:
```
# 4.Configure Promtail Deployment
For Promtail, make sure the following settings are added to the Packages Helm Chart [values.yaml](https://repo1.dso.mil/big-bang/product/packages/promtail/-/blob/main/chart/values.yaml) file:
```yaml
......@@ -73,6 +75,7 @@ vpa:
```
# 5.Verify VPA Status
Check the VPA status to ensure it is functioning correctly and providing recommendations.
Run the following command:
......@@ -84,7 +87,7 @@ Run the following command:
Look for the Conditions section and verify that the status is Healthy.
# 6.Monitor and Observe Scaling
Monitor the cluster and observe how the VPA scales the Fluent Bit and Promtail pods based on resource utilization.
Review the pod resource utilization metrics using commands like kubectl top pods or monitoring tools like Prometheus and Grafana.
By following these steps, you should be able to install and configure the Vertical Pod Autoscaler (VPA) to vertically scale Fluent Bit and Promtail pods in your Kubernetes cluster when additional resources are needed and headroom is available. Adjust the configurations and policies based on your specific requirements and application characteristics.
Monitor the cluster and observe how the VPA scales the Fluent Bit and Promtail pods based on resource utilization. Review the pod resource utilization metrics using commands like kubectl top pods or monitoring tools like Prometheus and Grafana.
By following these steps, you should be able to install and configure the VPA to vertically scale Fluent Bit and Promtail pods in your Kubernetes cluster when additional resources are needed and headroom is available. Adjust the configurations and policies based on your specific requirements and application characteristics.
# Credentials for Big Bang Packages
This document includes details on credentials to access each package in a default install (without SSO). It is safe to assume that any packages not listed in the two categories below either have no need for authentication or use different methods (ex: Velero require kubectl access).
This document includes details on credentials to access each package in a default install (i.e., without SSO). It is safe to assume that any packages not listed in the two categories below either have no need for authentication or use different methods (e.g., Velero require kubectl access).
## Packages With No Built in Authentication
## Packages With No Built-in Authentication
Although the below applications have no built in authentication, Big Bang's helm values can be configured to deploy authservice in front of these endpoints. Authservice is an Authentication Proxy that can integrate with SSO providers like Keycloak.
......@@ -10,9 +10,9 @@ Although the below applications have no built in authentication, Big Bang's helm
- Monitoring (Prometheus)
- Monitoring (Alertmanager)
## Packages With Built in Authentication
## Packages With Built-in Authentication
The applications in the table below provide both SSO and built in auth. The table gives default credentials and ways to access and/or override those.
The applications in the table below provide both SSO and built-in authentication. The table gives default credentials and ways to access and/or override those.
| Package (Application) | Default Username | Default Password | Additional Notes |
| --------------------- | ---------------- | ---------------- | ---------------- |
......
# Switching Between EFK and PLG
These instructions detail how to switch between EFK and PLG logging solutions.
These instructions detail how to switch between Elasticsearch, Fluentbit, and Kibana (EFK) and Promtail, Loki, and Grafana (PLG) logging solutions.
The EFK stack is an open-source choice for the Kubernetes log aggregation and analysis and is comprised of the following:
- Elasticsearch is a distributed and scalable search engine commonly used to sift through large volumes of log data.
- Fluentbit is a log shipper. It is an open source log collection agent which support multiple data sources and output formats.
- Kibana is a UI tool for querying, data visualization and dashboards.
- Kibana is a User Interface (UI) tool for querying, data visualization and dashboards.
Today the EFK stack (Elasticsearch, Fluentbit, and Kibana) is enabled by default in the bigbang chart.
The EFK stack appears within the chart as follows:
Today the EFK stack is enabled by default in the Big Bang chart. The EFK stack appears within the chart as shown in the following:
```yaml
......@@ -26,7 +25,8 @@ fluentbit:
```
If you want to use a logging solution that doesn't require a license and has a smaller footprint, Big Bang provides PLG. PLG is comprised of:
If you want to use a logging solution that doesn't require a license and has a smaller footprint, Big Bang provides PLG. PLG is comprised of the following:
- Promtail is an agent that detects targets (e.g., local log files), attaches labels to log streams from the pods, and ships them to Loki.
- Loki is an open-source, multi-tenant log aggregation system. It can be used with Grafana and Promtail to collect and access logs.
- Grafana is an open-source visualization platform that processes time-series data from Loki and makes the logs accessible in a web UI.
......@@ -65,5 +65,4 @@ tempo:
```
NOTE:
Both Fluentbit and Promtail forward logs to Grafana Loki. Should you want a logging stack comprised solely of Grafana technologies, the PLG stack will accommodate. The use of Fluentbit to send logs (log forwarder) is still a viable option rather than Promtail, and is configured within Big Bang to ship to Loki if enabled. Big Bang's recommendation is to use Fluentbit as a log forwarder because it is more feature rich. Promtail can only send to a limited set of endpoints (e.g., Loki and S3), whereas Fluentbit can send to numerous endpoints.
**NOTE:** Both Fluentbit and Promtail forward logs to Grafana Loki. Should you want a logging stack comprised solely of Grafana technologies, the PLG stack will accommodate. The use of Fluentbit to send logs (i.e., log forwarder) is still a viable option rather than Promtail, and is configured within Big Bang to ship to Loki if enabled. Big Bang's recommendation is to use Fluentbit as a log forwarder because it is more feature rich. Promtail can only send to a limited set of endpoints (e.g., Loki and S3), whereas Fluentbit can send to numerous endpoints.
......@@ -4,7 +4,7 @@
Big Bang is currently working to standardize the adoption of a global image pull policy so that customers can set a single value and have it passed to all packages.
The global image pull policy has been adopted in Big Bang for the core packages and for addons. In the Big Bang values.yaml file, a global parameter has been created to set the global image pull policy (`imagePullPolicy` in values) and it gets passed down to all core packages and addons spec. The default value for this global policy is `IfNotPresent`.
The global image pull policy has been adopted in Big Bang for the core packages and for addons. In the Big Bang values.yaml file, a global parameter has been created to set the global image pull policy (`imagePullPolicy` in values) and it gets passed down to all core packages and add-ons spec. The default value for this global policy is `IfNotPresent`.
We have also documented the package overrides required if you want to set a single package/pod with a different pull policy than the global.
......
# List Of Big Bang Applications That Are Using Kyverno
## Gitlab Runner
In order for Gitlab Runner auto registration to work the runner must be deployed in the `gitlab` namespace or have a copy of the 'gitlab-runner-secret' in its separate namespace. The Big Bang helm chart deploys Gitlab Runner in a separate namespace. Gitlab Runner uses Kyverno to copy the gitlab runner secret. There are alternate options if Kyverno is not wanted.
1. By default Gitlab Runner uses Kyverno to support auto registration
In order for Gitlab Runner auto registration to work, the runner must be deployed in the `gitlab` namespace or have a copy of the 'gitlab-runner-secret' in its separate namespace. The Big Bang helm chart deploys Gitlab Runner in a separate namespace. Gitlab Runner uses Kyverno to copy the gitlab runner secret. There are alternate options if Kyverno is not wanted.
1. By default Gitlab Runner uses Kyverno to support auto registration.
2. The runner token can be added to value overrides.
3. The gitlab-runner-secret can be manually copied to the `gitlab-runner` namespace.
## Fluentbit
In order for Fluentbit to automatically have a connection setup for for Elastic the Elastic root password and certificate secrets must be copied from the `logging` namespace to `fluentbit`. Kyverno is leveraged to copy these automatically. There are alternatives if using Kyverno is not desired:
In order for Fluentbit to automatically have a connection set up for Elastic, the Elastic root password and certificate secrets must be copied from the `logging` namespace to `fluentbit`. Kyverno is leveraged to copy these automatically. There are alternatives if using Kyverno is not desired. These alternatives are listed in the following:
1. By default Kyverno is used for auto-connection.
2. `fluentbit.values.additionalOutputs.elasticsearch` could be used to setup a connection with Elastic rather than the auto-connection.
3. Secrets can be manually copied to the `fluentbit` namespace.
# Integrating Prometheus Label Scraping for sample application
# Integrating Prometheus Label Scraping for Sample Application
## Introduction
Integrating Prometheus metrics scraping with label scraping services is helpful for monitoring monitoring applications that either don't ship with a a ServiceMonitor, for testing in researching to build out a serviceMonitor resource or in-development applications where creating and managing annotations is simpler than building and managing ServiceMonitors. This guide will explain how to integrate metrics scraping for your annotated service endpoints or specific pods with Big Bang.
Integrating Prometheus metrics scraping with label scraping services is helpful for monitoring monitoring applications that either don't ship with a a ServiceMonitor, for testing in researching to build out a serviceMonitor resource, or in-development applications where creating and managing annotations is simpler than building and managing ServiceMonitors. This guide will explain how to integrate metrics scraping for your annotated service endpoints or specific pods with Big Bang.
By default, global endpoint monitoring is behind a hidden value in Big Bang. If you wish to enable it the sections below are for each configuration.
By default, global endpoint monitoring is behind a hidden value in Big Bang. If you wish to enable, it the sections below are for each configuration.
The available Global serviceMonitor for annotated endpoints service is defined by the prometheus job name `kubernetes-service-endpoints` with following hidden value `monitoring.globalServiceEndpointMetrics` in `values.yaml`:
```
......@@ -33,7 +33,7 @@ Before integrating with Prometheus, ensure the following:
### 1. Define Placeholder Values for a Service (Recommended)
Add placeholders in `chart/values.yaml` to configure whether the monitoring stack (Prometheus) is enabled:
the following is an example of a placeholder for allowing scraping of the `metrics-server` service endpoint:
the following is an example of a placeholder for allowing scraping of the `metrics-server` service endpoint:
```yaml
addons:
metrics-server:
......@@ -46,10 +46,10 @@ addons:
serviceMonitor:
enabled: false # set to true to enable monitoring if the service is not already being scraped. This is an easily re-producible example but ideally the app will not ship with a serviceMonitor.
note: The example above is for the metrics-server application, you will need to update the service annotations to match your application. Also note that metrics-server is already being scraped by a local serviceMonitor, so you will need to disable it so label scraping can be enabled.
**NOTE:** The example above is for the metrics-server application, you will need to update the service annotations to match your application. Also note that metrics-server is already being scraped by a local serviceMonitor, so you will need to disable it so label scraping can be enabled.
```
Applications configured correctly will be scraped by Prometheus under the `target->` `kubernetes-service-endpoints` (see image below) in the Prometheus UI. If you do not know the correct port and path for your application, you may use `Service Discovery` in the Prometheus UI to find the correct port and path for your application.
Applications configured correctly will be scraped by Prometheus under the `target->` `kubernetes-service-endpoints` in the Prometheus User Interface (UI). Refer to the image below. If you do not know the correct port and path for your application, you may use `Service Discovery` in the Prometheus UI to find the correct port and path for your application.
![Prometeus GUI](../../assets/imgs/developer/metrics-server-scraping.png)
![Prometheus GUI](../../assets/imgs/developer/metrics-server-scraping.png)
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment