Loki Dashboard quick search not displaying logs

Loki log query appears broken. Loki Dashboard quick search not displaying logs for any namespace on dogfood. See Loki dashboard link on dogfood

From Kevin Wilder -> the loki strategy: scalable broke with 5.21.0-bb.X

Working logs with 5.15.0-bb.0 looks like this ->

added kindbug priority1 labels

set weight to 2

changed iteration to Big Bang Iterations Oct 3, 2023 - Oct 16, 2023

changed the description

see related issue created on bigbang repo
big-bang/bigbang#1799 (closed)

mentioned in issue big-bang/bigbang#1799 (closed)

assigned to @andrewshoell

changed milestone to %2.13.0

added loki teambigbang labels

added statusdoing label

marked this issue as related to big-bang/bigbang#1799 (closed)

changes found - https://repo1.dso.mil/big-bang/product/packages/loki/-/compare/5.15.0-bb.0...5.21.0-bb.3?from_project_id=7444&straight=true

created branch 68-loki-dashboard-quick-search-not-displaying-logs to address this issue

mentioned in merge request !109 (merged)

reproducable with latest including the following (in addition to the [logging-monitoring-values.yaml here]( big-bang/bigbang#1799 (closed))

# success
loki:
  enabled: true
  sourceType: "git"
  git:
    repo: "https://repo1.dso.mil/big-bang/product/packages/loki.git"
    path: "chart"
    tag: "5.15.0-bb.0"

# fail
loki:
  enabled: true
  sourceType: "git"
  git:
    repo: "https://repo1.dso.mil/big-bang/product/packages/loki.git"
    path: "chart"
    tag: "5.21.0-bb.3"

validated 5.21.0-bb.1 has this issue as well (5.21.0-bb.0 doesn't deploy because of an error in the upstream)

validated 5.23.1-bb.0 is broken as well. Investigating the raw data to see what is missing

@kevin.wilder you can use chart versions >= 5.21.-bb. with loki.strategy = scalable if you set loki.values.backend.replicas to >= 3. I will add a helm validation to the loki chart to have it fail hard with a more useful error message if that is misconfigured.

EDIT: loki.values.backend.replicas only needs to be >= 2

added statusreview label and removed statusdoing label

mentioned in merge request big-bang/bigbang!3285 (merged)

closed with merge request big-bang/bigbang!3285 (merged)

mentioned in commit big-bang/bigbang@da3c539c