Resolve "Configure alloy-logs to forward logs to Loki"

General MR

Summary

Configure Alloy to ship logs to Loki

NOTE: logging is disabled by default as there's no garantee Loki will be enabled; this will need to be enabled at the Umbrella separately.

• Added alloy-logs
• Added podLogs
• Disable unused features
  • applicationObservability
  • alloyReceiver
  • alloy-metrics is technically also unused, but needs to be enabled due to templating issue #90 (closed)
• Update Maintenance docs for alloy-logs testing
• Renamed destinations for more accurate endpoint
  • Will be necessary to support multiple endpoints, such as mimir and/or prometheus

Relevant logs/screenshots

(Include any relevant logs/screenshots)

Linked Issue

issue

Upgrade Notices

N/A

added kindfeature statusdoing teamObservability labels

assigned to @bjacksonfv

added 1 commit

• fdfdf686 - disable appObs and receiver by default - currently unused

Compare with previous version

added 1 commit

• 7bdca403 - Add protocol http to loki destination

Compare with previous version

changed the description

added 1 commit

• cd789395 - Update alloy testing instructions

Compare with previous version

added 1 commit

• 75f52f42 - Destination name change

Compare with previous version

changed the description

added 1 commit

• a77abcd7 - Update Network Policy to target all Alloy pods instead of just metrics

Compare with previous version

added 8 commits

• a77abcd7...fcfd8dab - 2 commits from branch main
• 7b0e21b2 - Enable alloy-logs, podLogs, set securityContext
• a09e2874 - move enabled to standard line
• 93f05a54 - disable appObs and receiver by default - currently unused
• dfff7d85 - Add protocol http to loki destination
• 750d79b6 - Update alloy testing instructions
• 2fe35437 - Destination name change

Compare with previous version

added 1 commit

• 456a37f9 - Chart bump, changelog, readme

Compare with previous version

added 1 commit

• 9120b5a5 - Disable podLogs by default; needs to be enabled by Umbrella

Compare with previous version

added 1 commit

• 76306e1f - Disable logging by default; needs to be enabled by Umbrella

Compare with previous version

changed the description

Testing

1. Checkout bigbang branch alloy-log-kyverno-exceptions
2. Deploy with following overrides

domain: dev.bigbang.mil                                                                                                   
                                                                                                                          
flux:                                                                                                                     
  interval: 1m                                                                                                            
  rollback:                                                                                                               
    cleanupOnFail: false                                                                                                  
                                                                                                                          
istio:                                                                                                                    
  enabled: true                                                                                                           
                                                                                                                          
promtail:                                                                                                                 
  enabled: false                                                                                                          
                                                                                                                          
fluentbit:                                                                                                                
  enabled: false                                                                                                          
                                                                                                                          
loki:                                                                                                                     
  enabled: true                                                                                                           
  git:                                                                                                                    
    tag: null                                                                                                             
    branch: loki-netpol-alloy                                                                                             
                                                                                                                          
addons:                                                                                                                   
  alloy:                                                                                                                  
    enabled: true                                                                                                         
    git:                                                                                                                  
      tag: null                                                                                                           
      branch: 88-alloy-logs                                                                                               
    values:                                                                                                               
      k8s-monitoring:                                                                                                     
        alloy-logs: 
          enabled: true                                                                                               
        podLogs: 
          enabled: true     

3. Login to Grafana, checkout loki dashboards - Ensure logs are populating.

added statusreview label and removed statusdoing label

confirmed pods are healthy, grafana looks good and pods are populating