Enable https by default for kubernetes-service-endpoints

Feature Request

Why

What is the use case for the feature you are requesting? What are you trying to solve?

Currently, as recommended by Kubecost documentation, deployment of a new node-exporter and ksm is disabled. Instead, we use the node-exporter and ksm instance provided by the monitoring suite.

The deployment of the Kubecost Prometheus sets up the jobs for scraping using Kubernetes service discovery and assumes non-tls endpoints. Due to this, and the Peer Authentication in the monitoring suite, there are issues with networking the scrape.

Proposed Solution

My team has developed a work around that creates a new Peer Authentication in monitoring that allows Permissive mTLS, attempting to resolve the networking issues, but this seems to also affect Monitoring Promtheus and its ability to scrape. In addition, this lowers security posture.

Another work around would have us put all the default scrape configs into our helm values, but this causes us to have to maintain these scrape configs long term, and compare changes with the helm chart every time it is updated.

A better solution would be to enable the https scheme in the default scrape config, and setup a default TLS config. This would allow us to mount our TLS certs in the same way that monitoring utilizes for Prometheus.

This could use a default location for the certs, or a pass through value in the helm chart.