Loki cypress is intermittently failing in the Smoke Test for Big Bang
For Big Bang MR for bumping elasticsearch-kibana packages the CI/CD smoke test will intermittently fail at the loki test. Rerunning the pipeline will sometimes pass without any issues (which may indicate a timeout issue).
Cypress test for Loki
The cypress test failure for the cypress test for loki for MR to bump to 1.18.0-bb.5 that fails at clean install but passes the upgrade but the cypress for loki passes on a rerun on the same pipeline.
Designs
Related merge requests 5
When these merge requests are accepted, this issue will be closed automatically.
Activity
added kindbug loki statusready-to-work teamObservability labels
added triage-priority label
assigned to @kliu
added statusdoing label and removed statusready-to-work label
Running test on debug for loki-cypress failure big-bang/bigbang!5167 (closed)
-
MR made by @blairbowden should have fixed this timeout.
To confirm, I ran 3 pipeline tests where the loki-cypress-test passed (pipeline shows error due to kyverno package failing, unrelated to loki cypress).
removed statusdoing label
added statusreview label
mentioned in merge request big-bang/bigbang!5167 (closed)
added statusready-to-work label and removed statusreview label
-
10/1 loki cypress test failed-https://repo1.dso.mil/big-bang/bigbang/-/jobs/38730043#L3312. Need to look at increasing timeout.
-
Looked at the artifact for the above pipeline kubectl_describes/namespaces/logging/Pods.yaml the pod
logging-loki-read-57f94458b8-524qrhad the following errorEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Warning PolicyViolation 16m kyverno-scan policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Warning PolicyViolation 17m kyverno-admission policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Normal Scheduled 17m default-scheduler Successfully assigned logging/logging-loki-read-57f94458b8-524qr to k3d-38730043-server-0 Warning PolicyViolation 7m3s kyverno-scan policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Warning PolicyViolation 7m43s kyverno-scan policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Warning PolicyViolation 17m kyverno-admission policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Warning PolicyViolation 8m2s kyverno-scan policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Warning PolicyViolation 8m17s kyverno-scan policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Warning PolicyViolation 8m37s kyverno-scan policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Warning PolicyViolation 8m54s kyverno-scan policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Warning PolicyViolation 9m9s kyverno-scan policy require-labels/check-for-labels fail: validation error: The pod is missing a required label. rule check-for-labels failed at path /metadata/labels/kyverno-policies-bbtest/required/ Normal Started 17m kubelet Started container istio-init Normal Created 17m kubelet Created container istio-init Normal Pulled 17m kubelet Container image "registry1.dso.mil/ironbank/opensource/istio/proxyv2:1.22.4" already present on machine Normal Pulled 17m kubelet Container image "registry1.dso.mil/ironbank/opensource/istio/proxyv2:1.22.4" already present on machine Normal Started 16m kubelet Started container istio-proxy Normal Created 16m kubelet Created container istio-proxy Normal Pulling 16m kubelet Pulling image "registry1.dso.mil/ironbank/opensource/grafana/loki:3.1.1" Normal Created 16m kubelet Created container loki Normal Pulled 16m kubelet Successfully pulled image "registry1.dso.mil/ironbank/opensource/grafana/loki:3.1.1" in 5.161s (5.161s including waiting). Image size: 83786234 bytes. Normal Started 16m kubelet Started container loki Warning Unhealthy 14m (x10 over 16m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503and the logs for that pod- https://repo1.dso.mil/big-bang/bigbang/-/jobs/38730043/artifacts/file/pod_logs/logging/logging-loki-read-57f94458b8-qkzcc.txt Had the following error in the logs
[pod/logging-loki-read-57f94458b8-qkzcc/loki] level=warn ts=2024-10-01T15:18:35.961653628Z caller=resolver.go:134 index-store=boltdb-shipper-2022-01-11 msg="IP address lookup yielded no results. No host found or no addresses found" host=logging-loki-backend-headless.logging.svc.cluster.local [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:18:35.964256259Z caller=resolver.go:87 index-store=tsdb-2023-08-01 msg="failed to lookup IP addresses" host=logging-loki-backend-headless.logging.svc.cluster.local err="lookup logging-loki-backend-headless.logging.svc.cluster.local on 172.20.0.10:53: no such host" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=warn ts=2024-10-01T15:18:35.964282842Z caller=resolver.go:134 index-store=tsdb-2023-08-01 msg="IP address lookup yielded no results. No host found or no addresses found" host=logging-loki-backend-headless.logging.svc.cluster.local [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:18:35.966947524Z caller=resolver.go:87 index-store=tsdb-2024-05-30 msg="failed to lookup IP addresses" host=logging-loki-backend-headless.logging.svc.cluster.local err="lookup logging-loki-backend-headless.logging.svc.cluster.local on 172.20.0.10:53: no such host" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=warn ts=2024-10-01T15:18:35.966966804Z caller=resolver.go:134 index-store=tsdb-2024-05-30 msg="IP address lookup yielded no results. No host found or no addresses found" host=logging-loki-backend-headless.logging.svc.cluster.local [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:20:20.960948035Z caller=dns.go:78 index-store=boltdb-shipper-2022-01-11 msg="failed to resolve server addresses" err="lookup IP addresses \"logging-loki-backend-headless.logging.svc.cluster.local\": lookup logging-loki-backend-headless.logging.svc.cluster.local: i/o timeout" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:20:20.961913999Z caller=dns.go:78 index-store=tsdb-2023-08-01 msg="failed to resolve server addresses" err="lookup IP addresses \"logging-loki-backend-headless.logging.svc.cluster.local\": lookup logging-loki-backend-headless.logging.svc.cluster.local: i/o timeout" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:23:15.960957221Z caller=dns.go:78 index-store=boltdb-shipper-2022-01-11 msg="failed to resolve server addresses" err="lookup IP addresses \"logging-loki-backend-headless.logging.svc.cluster.local\": lookup logging-loki-backend-headless.logging.svc.cluster.local: i/o timeout" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:23:15.962754657Z caller=dns.go:78 index-store=tsdb-2023-08-01 msg="failed to resolve server addresses" err="lookup IP addresses \"logging-loki-backend-headless.logging.svc.cluster.local\": lookup logging-loki-backend-headless.logging.svc.cluster.local: i/o timeout" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=info ts=2024-10-01T15:26:18.349710625Z caller=roundtrip.go:435 org_id=fake msg="executing query" type=instant query=vector(1)+vector(1) query_hash=2536976802 [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:26:40.63820826Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query=vector(1)+vector(1) query_hash=2536976802 start=2096-10-02T07:06:40Z end=2096-10-02T07:06:40Z start_delta=-631167h39m59.361797721s end_delta=-631167h39m59.361797149s length=0s retry_in=363.308603ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=info ts=2024-10-01T15:27:18.61936223Z caller=roundtrip.go:376 org_id=fake msg="executing query" type=range query="sum(count_over_time({namespace=\"default\", pod=~\".+\"} |~ \"\"[2s]))" start=2024-10-01T14:57:18Z end=2024-10-01T15:27:18.523Z start_delta=30m0.619358787s end_delta=96.359069ms length=30m0.523s step=2000 query_hash=1022651721 [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:27:48.558306126Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T14:59:58Z end=2024-10-01T15:14:58Z start_delta=27m50.558301596s end_delta=12m50.55830211s length=15m0s retry_in=287.157086ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:27:48.55830266Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T15:14:58Z end=2024-10-01T15:27:18Z start_delta=12m50.558296801s end_delta=30.55829738s length=12m20s retry_in=321.547142ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:27:48.558355389Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T14:57:16Z end=2024-10-01T14:59:58Z start_delta=30m32.558354192s end_delta=27m50.558354307s length=2m42s retry_in=494.035403ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:27:48.558370896Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="sum(count_over_time({namespace=\"default\", pod=~\".+\"} |~ \"\"[2s]))" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:27:48.558373984Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="sum(count_over_time({namespace=\"default\", pod=~\".+\"} |~ \"\"[2s]))" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:27:48.55839172Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="sum(count_over_time({namespace=\"default\", pod=~\".+\"} |~ \"\"[2s]))" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=info ts=2024-10-01T15:27:51.601632239Z caller=roundtrip.go:376 org_id=fake msg="executing query" type=range query="sum(count_over_time({namespace=\"default\", pod=~\".+\"} |~ \"\"[2s]))" start=2024-10-01T14:57:50Z end=2024-10-01T15:27:51.516Z start_delta=30m1.60162878s end_delta=85.629021ms length=30m1.516s step=2000 query_hash=1022651721 [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:28:21.510330768Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T14:59:58Z end=2024-10-01T15:14:58Z start_delta=28m23.510325649s end_delta=13m23.510326104s length=15m0s retry_in=345.724968ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:28:21.510327781Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T15:14:58Z end=2024-10-01T15:27:50Z start_delta=13m23.510321971s end_delta=31.510322535s length=12m52s retry_in=370.611071ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:28:21.510341647Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T14:57:48Z end=2024-10-01T14:59:58Z start_delta=30m33.51033739s end_delta=28m23.510337702s length=2m10s retry_in=458.645124ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:28:21.510382505Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="sum(count_over_time({namespace=\"default\", pod=~\".+\"} |~ \"\"[2s]))" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:28:21.510381922Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="sum(count_over_time({namespace=\"default\", pod=~\".+\"} |~ \"\"[2s]))" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:28:21.510381944Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="sum(count_over_time({namespace=\"default\", pod=~\".+\"} |~ \"\"[2s]))" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=info ts=2024-10-01T15:28:24.706803732Z caller=roundtrip.go:376 org_id=fake msg="executing query" type=range query="{namespace=\"default\", pod=~\".+\"} |~ \"\"" start=2024-10-01T14:58:24.616Z end=2024-10-01T15:28:24.616Z start_delta=30m0.090799447s end_delta=90.79973ms length=30m0s step=2000 query_hash=1714547557 [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:28:54.598014505Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T14:57:54Z end=2024-10-01T15:00:00Z start_delta=31m0.598010666s end_delta=28m54.598010988s length=2m6s retry_in=250.973284ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:28:54.598006414Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T15:14:30Z end=2024-10-01T15:28:24Z start_delta=14m24.59800177s end_delta=30.598002301s length=13m54s retry_in=351.818392ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] level=error ts=2024-10-01T15:28:54.598001206Z caller=retry.go:95 org_id=fake msg="error processing request" try=0 query="{namespace=\"default\", pod=~\".+\"}" query_hash=2710986727 start=2024-10-01T14:59:30Z end=2024-10-01T15:15:00Z start_delta=29m24.597995199s end_delta=13m54.597995755s length=15m30s retry_in=487.136094ms err="context canceled" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:28:54.598061001Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="{namespace=\"default\", pod=~\".+\"} |~ \"\"" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:28:54.598061139Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="{namespace=\"default\", pod=~\".+\"} |~ \"\"" [pod/logging-loki-read-57f94458b8-qkzcc/loki] ts=2024-10-01T15:28:54.598063487Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware org_id=fake user=fake level=warn msg="failed mapping AST" err="context canceled" query="{namespace=\"default\", pod=~\".+\"} |~ \"\""Edited by Kirby Liu
added statusdoing label and removed statusready-to-work label
-
Using this draft mr for testing timeout big-bang/bigbang!5185 (closed)
mentioned in merge request big-bang/bigbang!5185 (closed)
-
I took a log of the
loki-cypress-testpod from https://repo1.dso.mil/big-bang/bigbang/-/jobs/38797825/artifacts/file/pod_logs/logging/loki-cypress-test.txterror: previous terminated container "istio-init" in pod "loki-cypress-test" not found error: previous terminated container "istio-proxy" in pod "loki-cypress-test" not found error: previous terminated container "loki-cypress-test" in pod "loki-cypress-test" not found [pod/loki-cypress-test/loki-cypress-test] --2024-10-03 20:16:03-- https://repo1.dso.mil/big-bang/product/packages/gluon/-/raw/master/common/commands.js [pod/loki-cypress-test/loki-cypress-test] Resolving repo1.dso.mil (repo1.dso.mil)... 15.205.173.153 [pod/loki-cypress-test/loki-cypress-test] Connecting to repo1.dso.mil (repo1.dso.mil)|15.205.173.153|:443... connected. [pod/loki-cypress-test/loki-cypress-test] HTTP request sent, awaiting response... 200 OK [pod/loki-cypress-test/loki-cypress-test] Length: 4156 (4.1K) [text/plain] [pod/loki-cypress-test/loki-cypress-test] Saving to: '/test/cypress/common/commands.js' [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] 0K .... 100% 291M=0s [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] 2024-10-03 20:16:03 (291 MB/s) - '/test/cypress/common/commands.js' saved [4156/4156] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] DevTools listening on ws://127.0.0.1:35571/devtools/browser/32c724d3-3d2b-4f9f-843f-d19a6e5c8085 [pod/loki-cypress-test/loki-cypress-test] This folder is not writable: /test [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] Writing to this directory is required by Cypress in order to store screenshots and videos. [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] Enable write permissions to this directory to ensure screenshots and videos are stored. [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] If you don't require screenshots or videos to be stored you can safely ignore this warning. [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] tput: No value for $TERM and no -T specified [pod/loki-cypress-test/loki-cypress-test] ================================================================================ [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] (Run Starting) [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ [pod/loki-cypress-test/loki-cypress-test] │ Cypress: 13.14.2 │ [pod/loki-cypress-test/loki-cypress-test] │ Browser: Chrome 129 (headless) │ [pod/loki-cypress-test/loki-cypress-test] │ Node Version: v20.16.0 (/usr/local/bin/node) │ [pod/loki-cypress-test/loki-cypress-test] │ Specs: 1 found (01-loki-health.spec.cy.js) │ [pod/loki-cypress-test/loki-cypress-test] │ Searched: cypress/e2e/**/*.cy.{js,jsx,ts,tsx} │ [pod/loki-cypress-test/loki-cypress-test] └────────────────────────────────────────────────────────────────────────────────────────────────┘ [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] ──────────────────────────────────────────────────────────────────────────────────────────────────── [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] Running: 01-loki-health.spec.cy.js (1 of 1) [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] Loki Test [pod/loki-cypress-test/loki-cypress-test] (Attempt 1 of 3) Check Loki is available as a data source in grafana and connected [pod/loki-cypress-test/loki-cypress-test] (Attempt 2 of 3) Check Loki is available as a data source in grafana and connected [pod/loki-cypress-test/loki-cypress-test] 1) Check Loki is available as a data source in grafana and connected [pod/loki-cypress-test/loki-cypress-test] (Attempt 1 of 3) Test for Loki Dashboard log data [pod/loki-cypress-test/loki-cypress-test] (Attempt 2 of 3) Test for Loki Dashboard log data [pod/loki-cypress-test/loki-cypress-test] 2) Test for Loki Dashboard log data [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] 0 passing (3m) [pod/loki-cypress-test/loki-cypress-test] 2 failing [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] 1) Loki Test [pod/loki-cypress-test/loki-cypress-test] Check Loki is available as a data source in grafana and connected: [pod/loki-cypress-test/loki-cypress-test] AssertionError: Timed out retrying after 12000ms: Expected to find element: `input[name="user"]`, but never found it. [pod/loki-cypress-test/loki-cypress-test] at Context.eval (webpack:///./cypress/common/commands.js:26:5) [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] 2) Loki Test [pod/loki-cypress-test/loki-cypress-test] Test for Loki Dashboard log data: [pod/loki-cypress-test/loki-cypress-test] CypressError: Timed out retrying after 30000ms: `cy.wait()` timed out waiting `30000ms` for the 1st response to the route: `apiQuery`. No response ever occurred. [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] https://on.cypress.io/wait [pod/loki-cypress-test/loki-cypress-test] at cypressErr (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:76025:18) [pod/loki-cypress-test/loki-cypress-test] at Object.errByPath (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:76079:10) [pod/loki-cypress-test/loki-cypress-test] at checkForXhr (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:135700:84) [pod/loki-cypress-test/loki-cypress-test] at <unknown> (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:135726:28) [pod/loki-cypress-test/loki-cypress-test] at tryCatcher (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:1807:23) [pod/loki-cypress-test/loki-cypress-test] at Promise.attempt.Promise.try (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:4315:29) [pod/loki-cypress-test/loki-cypress-test] at whenStable (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:144102:68) [pod/loki-cypress-test/loki-cypress-test] at <unknown> (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:144043:14) [pod/loki-cypress-test/loki-cypress-test] at tryCatcher (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:1807:23) [pod/loki-cypress-test/loki-cypress-test] at Promise._settlePromiseFromHandler (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:1519:31) [pod/loki-cypress-test/loki-cypress-test] at Promise._settlePromise (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:1576:18) [pod/loki-cypress-test/loki-cypress-test] at Promise._settlePromise0 (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:1621:10) [pod/loki-cypress-test/loki-cypress-test] at Promise._settlePromises (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:1701:18) [pod/loki-cypress-test/loki-cypress-test] at Promise._fulfill (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:1645:18) [pod/loki-cypress-test/loki-cypress-test] at <unknown> (https://grafana.dev.bigbang.mil/__cypress/runner/cypress_runner.js:5450:46) [pod/loki-cypress-test/loki-cypress-test] From Your Spec Code: [pod/loki-cypress-test/loki-cypress-test] at Context.eval (webpack:///./cypress/common/commands.js:89:5) [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] (Results) [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ [pod/loki-cypress-test/loki-cypress-test] │ Tests: 2 │ [pod/loki-cypress-test/loki-cypress-test] │ Passing: 0 │ [pod/loki-cypress-test/loki-cypress-test] │ Failing: 2 │ [pod/loki-cypress-test/loki-cypress-test] │ Pending: 0 │ [pod/loki-cypress-test/loki-cypress-test] │ Skipped: 0 │ [pod/loki-cypress-test/loki-cypress-test] │ Screenshots: 6 │ [pod/loki-cypress-test/loki-cypress-test] │ Video: true │ [pod/loki-cypress-test/loki-cypress-test] │ Duration: 3 minutes, 4 seconds │ [pod/loki-cypress-test/loki-cypress-test] │ Spec Ran: 01-loki-health.spec.cy.js │ [pod/loki-cypress-test/loki-cypress-test] └────────────────────────────────────────────────────────────────────────────────────────────────┘ [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] (Screenshots) [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] - /test/cypress/screenshots/01-loki-health.spec.cy.js/Loki Test -- Check Loki is a (1280x581) [pod/loki-cypress-test/loki-cypress-test] vailable as a data source in grafana and connected (failed).png [pod/loki-cypress-test/loki-cypress-test] - /test/cypress/screenshots/01-loki-health.spec.cy.js/Loki Test -- Check Loki is a (1280x581) [pod/loki-cypress-test/loki-cypress-test] vailable as a data source in grafana and connected (failed) (attempt 2).png [pod/loki-cypress-test/loki-cypress-test] - /test/cypress/screenshots/01-loki-health.spec.cy.js/Loki Test -- Check Loki is a (1280x581) [pod/loki-cypress-test/loki-cypress-test] vailable as a data source in grafana and connected (failed) (attempt 3).png [pod/loki-cypress-test/loki-cypress-test] - /test/cypress/screenshots/01-loki-health.spec.cy.js/Loki Test -- Test for Loki D (1280x581) [pod/loki-cypress-test/loki-cypress-test] ashboard log data (failed).png [pod/loki-cypress-test/loki-cypress-test] - /test/cypress/screenshots/01-loki-health.spec.cy.js/Loki Test -- Test for Loki D (1280x581) [pod/loki-cypress-test/loki-cypress-test] ashboard log data (failed) (attempt 2).png [pod/loki-cypress-test/loki-cypress-test] - /test/cypress/screenshots/01-loki-health.spec.cy.js/Loki Test -- Test for Loki D (1280x581) [pod/loki-cypress-test/loki-cypress-test] ashboard log data (failed) (attempt 3).png [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] (Video) [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] - Started compressing: Compressing to 35 CRF [pod/loki-cypress-test/loki-cypress-test] Compression progress: 98% [pod/loki-cypress-test/loki-cypress-test] - Finished compressing: 12 seconds [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] - Video output: /test/cypress/videos/01-loki-health.spec.cy.js.mp4 [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] tput: No value for $TERM and no -T specified [pod/loki-cypress-test/loki-cypress-test] ================================================================================ [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] (Run Finished) [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] Spec Tests Passing Failing Pending Skipped [pod/loki-cypress-test/loki-cypress-test] ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ [pod/loki-cypress-test/loki-cypress-test] │ ✖ 01-loki-health.spec.cy.js 03:04 2 - 2 - - │ [pod/loki-cypress-test/loki-cypress-test] └────────────────────────────────────────────────────────────────────────────────────────────────┘ [pod/loki-cypress-test/loki-cypress-test] ✖ 1 of 1 failed (100%) 03:04 2 - 2 - - [pod/loki-cypress-test/loki-cypress-test] [pod/loki-cypress-test/loki-cypress-test] npm notice [pod/loki-cypress-test/loki-cypress-test] npm notice New minor version of npm available! 10.8.1 -> 10.9.0 [pod/loki-cypress-test/loki-cypress-test] npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.9.0 [pod/loki-cypress-test/loki-cypress-test] npm notice To update run: npm install -g npm@10.9.0 [pod/loki-cypress-test/loki-cypress-test] npm notice [pod/loki-cypress-test/loki-cypress-test] OK [pod/loki-cypress-test/istio-init] 2024-10-03T20:15:59.638380Z info Istio iptables environment: [pod/loki-cypress-test/istio-init] ENVOY_PORT= [pod/loki-cypress-test/istio-init] INBOUND_CAPTURE_PORT= [pod/loki-cypress-test/istio-init] ISTIO_INBOUND_INTERCEPTION_MODE= [pod/loki-cypress-test/istio-init] ISTIO_INBOUND_TPROXY_ROUTE_TABLE= [pod/loki-cypress-test/istio-init] ISTIO_INBOUND_PORTS= [pod/loki-cypress-test/istio-init] ISTIO_OUTBOUND_PORTS= [pod/loki-cypress-test/istio-init] ISTIO_LOCAL_EXCLUDE_PORTS= [pod/loki-cypress-test/istio-init] ISTIO_EXCLUDE_INTERFACES= [pod/loki-cypress-test/istio-init] ISTIO_SERVICE_CIDR= [pod/loki-cypress-test/istio-init] ISTIO_SERVICE_EXCLUDE_CIDR= [pod/loki-cypress-test/istio-init] ISTIO_META_DNS_CAPTURE= [pod/loki-cypress-test/istio-init] INVALID_DROP= [pod/loki-cypress-test/istio-init] [pod/loki-cypress-test/istio-init] 2024-10-03T20:15:59.638428Z info Istio iptables variables: [pod/loki-cypress-test/istio-init] PROXY_PORT=15001 [pod/loki-cypress-test/istio-init] PROXY_INBOUND_CAPTURE_PORT=15006 [pod/loki-cypress-test/istio-init] PROXY_TUNNEL_PORT=15008 [pod/loki-cypress-test/istio-init] PROXY_UID=1337 [pod/loki-cypress-test/istio-init] PROXY_GID=1337 [pod/loki-cypress-test/istio-init] INBOUND_INTERCEPTION_MODE=REDIRECT [pod/loki-cypress-test/istio-init] INBOUND_TPROXY_MARK=1337 [pod/loki-cypress-test/istio-init] INBOUND_TPROXY_ROUTE_TABLE=133 [pod/loki-cypress-test/istio-init] INBOUND_PORTS_INCLUDE=* [pod/loki-cypress-test/istio-init] INBOUND_PORTS_EXCLUDE=15090,15021,15020 [pod/loki-cypress-test/istio-init] OUTBOUND_OWNER_GROUPS_INCLUDE=* [pod/loki-cypress-test/istio-init] OUTBOUND_OWNER_GROUPS_EXCLUDE= [pod/loki-cypress-test/istio-init] OUTBOUND_IP_RANGES_INCLUDE=* [pod/loki-cypress-test/istio-init] OUTBOUND_IP_RANGES_EXCLUDE= [pod/loki-cypress-test/istio-init] OUTBOUND_PORTS_INCLUDE= [pod/loki-cypress-test/istio-init] OUTBOUND_PORTS_EXCLUDE= [pod/loki-cypress-test/istio-init] KUBE_VIRT_INTERFACES= [pod/loki-cypress-test/istio-init] ENABLE_INBOUND_IPV6=false [pod/loki-cypress-test/istio-init] DUAL_STACK=false [pod/loki-cypress-test/istio-init] DNS_CAPTURE=false [pod/loki-cypress-test/istio-init] DROP_INVALID=false [pod/loki-cypress-test/istio-init] CAPTURE_ALL_DNS=false [pod/loki-cypress-test/istio-init] DNS_SERVERS=[],[] [pod/loki-cypress-test/istio-init] NETWORK_NAMESPACE= [pod/loki-cypress-test/istio-init] CNI_MODE=false [pod/loki-cypress-test/istio-init] EXCLUDE_INTERFACES= [pod/loki-cypress-test/istio-init] [pod/loki-cypress-test/istio-init] 2024-10-03T20:15:59.638485Z info Running iptables restore with: iptables-restore and the following input: [pod/loki-cypress-test/istio-init] * nat [pod/loki-cypress-test/istio-init] -N ISTIO_INBOUND [pod/loki-cypress-test/istio-init] -N ISTIO_REDIRECT [pod/loki-cypress-test/istio-init] -N ISTIO_IN_REDIRECT [pod/loki-cypress-test/istio-init] -N ISTIO_OUTPUT [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001 [pod/loki-cypress-test/istio-init] -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006 [pod/loki-cypress-test/istio-init] -A PREROUTING -p tcp -j ISTIO_INBOUND [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT [pod/loki-cypress-test/istio-init] -A OUTPUT -p tcp -j ISTIO_OUTPUT [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -j ISTIO_REDIRECT [pod/loki-cypress-test/istio-init] COMMIT [pod/loki-cypress-test/istio-init] 2024-10-03T20:15:59.638523Z info Running command (with wait lock): iptables-restore --noflush --wait=30 [pod/loki-cypress-test/istio-init] 2024-10-03T20:16:00.133307Z info Running iptables restore with: ip6tables-restore and the following input: [pod/loki-cypress-test/istio-init] [pod/loki-cypress-test/istio-init] 2024-10-03T20:16:00.133410Z info Running command (with wait lock): ip6tables-restore --noflush --wait=30 [pod/loki-cypress-test/istio-init] 2024-10-03T20:16:00.134613Z info Running command (without lock): iptables-save [pod/loki-cypress-test/istio-init] 2024-10-03T20:16:00.136286Z info Command output: [pod/loki-cypress-test/istio-init] # Generated by iptables-save v1.8.9 on Thu Oct 3 20:16:00 2024 [pod/loki-cypress-test/istio-init] *nat [pod/loki-cypress-test/istio-init] :PREROUTING ACCEPT [0:0] [pod/loki-cypress-test/istio-init] :INPUT ACCEPT [0:0] [pod/loki-cypress-test/istio-init] :OUTPUT ACCEPT [0:0] [pod/loki-cypress-test/istio-init] :POSTROUTING ACCEPT [0:0] [pod/loki-cypress-test/istio-init] :ISTIO_INBOUND - [0:0] [pod/loki-cypress-test/istio-init] :ISTIO_IN_REDIRECT - [0:0] [pod/loki-cypress-test/istio-init] :ISTIO_OUTPUT - [0:0] [pod/loki-cypress-test/istio-init] :ISTIO_REDIRECT - [0:0] [pod/loki-cypress-test/istio-init] -A PREROUTING -p tcp -j ISTIO_INBOUND [pod/loki-cypress-test/istio-init] -A OUTPUT -p tcp -j ISTIO_OUTPUT [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT [pod/loki-cypress-test/istio-init] -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006 [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN [pod/loki-cypress-test/istio-init] -A ISTIO_OUTPUT -j ISTIO_REDIRECT [pod/loki-cypress-test/istio-init] -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001 [pod/loki-cypress-test/istio-init] COMMIT [pod/loki-cypress-test/istio-init] # Completed on Thu Oct 3 20:16:00 2024 [pod/loki-cypress-test/istio-init] [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222474Z info FLAG: --concurrency="0" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222494Z info FLAG: --domain="logging.svc.cluster.local" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222499Z info FLAG: --help="false" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222502Z info FLAG: --log_as_json="false" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222508Z info FLAG: --log_caller="" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222511Z info FLAG: --log_output_level="default:info" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222514Z info FLAG: --log_rotate="" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222516Z info FLAG: --log_rotate_max_age="30" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222521Z info FLAG: --log_rotate_max_backups="1000" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222524Z info FLAG: --log_rotate_max_size="104857600" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222527Z info FLAG: --log_stacktrace_level="default:none" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222532Z info FLAG: --log_target="[stdout]" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222549Z info FLAG: --meshConfig="./etc/istio/config/mesh" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222555Z info FLAG: --outlierLogPath="" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222557Z info FLAG: --profiling="true" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222560Z info FLAG: --proxyComponentLogLevel="misc:error" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222562Z info FLAG: --proxyLogLevel="warning" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222567Z info FLAG: --serviceCluster="istio-proxy" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222570Z info FLAG: --stsPort="0" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222573Z info FLAG: --templateFile="" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222575Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222579Z info FLAG: --vklog="0" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222584Z info Version 1.22.5-f57a44a5bcbbba4cae93c872c30e361592a2129f-Clean [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222590Z info Set max file descriptors (ulimit -n) to: 65535 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222878Z info Proxy role ips=[172.21.3.78] type=sidecar id=loki-cypress-test.logging domain=logging.svc.cluster.local [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.222938Z info Apply proxy config from env {"tracing":{"zipkin":{"address":"tempo-tempo.tempo.svc:9411"},"sampling":100},"holdApplicationUntilProxyStarts":true} [pod/loki-cypress-test/istio-proxy] [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.226532Z info cpu limit detected as 1, setting concurrency [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.227040Z info Effective config: binaryPath: /usr/local/bin/envoy [pod/loki-cypress-test/istio-proxy] concurrency: 1 [pod/loki-cypress-test/istio-proxy] configPath: ./etc/istio/proxy [pod/loki-cypress-test/istio-proxy] controlPlaneAuthPolicy: MUTUAL_TLS [pod/loki-cypress-test/istio-proxy] discoveryAddress: istiod.istio-system.svc:15012 [pod/loki-cypress-test/istio-proxy] drainDuration: 45s [pod/loki-cypress-test/istio-proxy] holdApplicationUntilProxyStarts: true [pod/loki-cypress-test/istio-proxy] proxyAdminPort: 15000 [pod/loki-cypress-test/istio-proxy] serviceCluster: istio-proxy [pod/loki-cypress-test/istio-proxy] statNameLength: 189 [pod/loki-cypress-test/istio-proxy] statusPort: 15020 [pod/loki-cypress-test/istio-proxy] terminationDrainDuration: 5s [pod/loki-cypress-test/istio-proxy] tracing: [pod/loki-cypress-test/istio-proxy] sampling: 100 [pod/loki-cypress-test/istio-proxy] zipkin: [pod/loki-cypress-test/istio-proxy] address: tempo-tempo.tempo.svc:9411 [pod/loki-cypress-test/istio-proxy] [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.227053Z info JWT policy is third-party-jwt [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.227058Z info using credential fetcher of JWT type in cluster.local trust domain [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.331380Z info Workload SDS socket not found. Starting Istio SDS Server [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.331406Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.331425Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.331580Z info Opening status port 15020 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.339450Z info ads All caches have been synced up in 117.648752ms, marking server ready [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.339674Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.341512Z info Pilot SAN: [istiod.istio-system.svc] [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.345041Z info Starting proxy agent [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.345095Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --concurrency 1] [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.370508Z info sds Starting SDS grpc server [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.370538Z info sds Starting SDS server for workload certificates, will listen on "./var/run/secrets/workload-spiffe-uds/socket" [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.370661Z info starting Http service at 127.0.0.1:15004 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.451292Z warning envoy main external/envoy/source/server/server.cc:928 There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor. thread=11 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.452818Z warning envoy main external/envoy/source/server/server.cc:835 Usage of the deprecated runtime key overload.global_downstream_max_connections, consider switching to `envoy.resource_monitors.downstream_connections` instead.This runtime key will be removed in future. thread=11 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.538201Z info cache generated new workload certificate resourceName=default latency=179.884543ms ttl=23h59m59.461811085s [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.538242Z info cache Root cert has changed, start rotating root cert [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.538262Z info ads XDS: Incremental Pushing ConnectedEndpoints:0 Version: [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.538321Z info cache returned workload trust anchor from cache ttl=23h59m59.461680541s [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.538585Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=1 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.621111Z info ads ADS: new connection for node:loki-cypress-test.logging-1 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.621206Z info cache returned workload certificate from cache ttl=23h59m59.37879533s [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.621519Z info ads SDS: PUSH request for node:loki-cypress-test.logging resources:1 size:4.0kB resource:default [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.622751Z info ads ADS: new connection for node:loki-cypress-test.logging-2 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.622814Z info cache returned workload trust anchor from cache ttl=23h59m59.377187711s [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:01.622940Z info ads SDS: PUSH request for node:loki-cypress-test.logging resources:1 size:1.1kB resource:ROOTCA [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:02.677325Z info Readiness succeeded in 1.460222068s [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:16:02.677762Z info Envoy proxy is ready [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:03.197Z] "- - -" 0 - - - "-" 808 10739 133 - "-" "-" "-" "-" "15.205.173.153:443" outbound|443||repo1.dso.mil 172.21.3.78:52924 15.205.173.153:443 172.21.3.78:52910 repo1.dso.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:09.061Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 142.250.189.234:443 172.21.3.78:42040 safebrowsingohttpgateway.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:18.704Z] "- - -" 0 - - - "-" 2530 42051 2286 - "-" "-" "-" "-" "142.251.46.234:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:57302 142.251.32.42:443 172.21.3.78:45416 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:21.539Z] "- - -" 0 - - - "-" 1099 18917 214 - "-" "-" "-" "-" "142.250.191.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:37436 172.217.164.106:443 172.21.3.78:57318 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:21.650Z] "- - -" 0 - - - "-" 1099 20315 219 - "-" "-" "-" "-" "172.217.12.106:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:55062 142.251.46.234:443 172.21.3.78:57312 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:21.649Z] "- - -" 0 - - - "-" 1605 29795 1552 - "-" "-" "-" "-" "142.251.32.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:45422 142.250.189.170:443 172.21.3.78:53562 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:20.851Z] "- - -" 0 - - - "-" 3111 62310 12502 - "-" "-" "-" "-" "142.250.189.170:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:53548 172.217.12.106:443 172.21.3.78:55060 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:33.228Z] "- - -" 0 - - - "-" 1096 20315 211 - "-" "-" "-" "-" "142.250.191.74:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:45718 142.251.46.170:443 172.21.3.78:43496 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:31.280Z] "- - -" 0 - - - "-" 4155 1304050 3483 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:33892 172.20.1.241:443 172.21.3.78:33886 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:34.736Z] "- - -" 0 - - - "-" 887 10483 86 - "-" "-" "-" "-" "34.120.177.193:443" outbound|443||grafana.com 172.21.3.78:44364 34.120.177.193:443 172.21.3.78:44352 grafana.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:35.702Z] "- - -" 0 - - - "-" 1100 20315 239 - "-" "-" "-" "-" "142.251.46.234:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:46440 172.217.164.106:443 172.21.3.78:49808 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:36.365Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 142.250.191.42:443 172.21.3.78:41890 passwordsleakcheck-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:36.177Z] "- - -" 0 - - - "-" 1086 20315 315 - "-" "-" "-" "-" "172.217.12.106:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:43380 142.251.46.202:443 172.21.3.78:34972 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:36.418Z] "- - -" 0 - - - "-" 1086 20315 286 - "-" "-" "-" "-" "142.251.32.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:59614 142.250.189.170:443 172.21.3.78:58134 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:36.760Z] "- - -" 0 - - - "-" 1086 20315 167 - "-" "-" "-" "-" "142.251.214.138:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:59956 142.250.189.170:443 172.21.3.78:58136 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:35.704Z] "- - -" 0 - - - "-" 1086 20315 1274 - "-" "-" "-" "-" "142.251.46.170:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:43526 142.251.32.42:443 172.21.3.78:59596 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:37.582Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 74.125.5.202:443 172.21.3.78:57366 r5---sn-nx5s7n7y.gvt1.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:36.176Z] "- - -" 0 - - - "-" 1086 20315 1631 - "-" "-" "-" "-" "142.250.191.74:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:45734 142.251.46.234:443 172.21.3.78:46444 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:37.941Z] "- - -" 0 - - - "-" 1115 20315 176 - "-" "-" "-" "-" "142.251.46.202:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:34998 142.250.191.74:443 172.21.3.78:45760 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:38.482Z] "- - -" 0 - - - "-" 1115 20315 284 - "-" "-" "-" "-" "142.250.191.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:58436 172.217.12.106:443 172.21.3.78:35456 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:38.694Z] "- - -" 0 - - - "-" 1115 20315 182 - "-" "-" "-" "-" "142.250.188.10:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:34688 172.217.164.106:443 172.21.3.78:51132 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:38.915Z] "- - -" 0 - - - "-" 1100 20315 160 - "-" "-" "-" "-" "142.251.46.170:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:59720 172.217.12.106:443 172.21.3.78:35468 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:38.967Z] "- - -" 0 - - - "-" 1115 20315 269 - "-" "-" "-" "-" "142.250.191.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:58438 142.250.189.202:443 172.21.3.78:33346 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:39.795Z] "- - -" 0 - - - "-" 1115 18917 227 - "-" "-" "-" "-" "142.251.46.170:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:59728 172.217.164.106:443 172.21.3.78:51144 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:55.429Z] "- - -" 0 - - - "-" 1100 20315 268 - "-" "-" "-" "-" "142.250.189.202:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:59010 142.251.214.138:443 172.21.3.78:46602 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:57.937Z] "- - -" 0 - - - "-" 1100 20315 183 - "-" "-" "-" "-" "142.251.32.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:40832 142.250.191.74:443 172.21.3.78:48014 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:35.683Z] "- - -" 0 - - - "-" 6302 269436 22725 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:33912 172.20.1.241:443 172.21.3.78:33898 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:31.246Z] "- - -" 0 - - - "-" 5920 502909 27190 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:33874 172.20.1.241:443 172.21.3.78:33838 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:00.109Z] "- - -" 0 - - - "-" 1101 20315 604 - "-" "-" "-" "-" "142.250.189.234:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:35238 142.251.32.42:443 172.21.3.78:38832 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:00.401Z] "- - -" 0 - - - "-" 1101 20315 328 - "-" "-" "-" "-" "142.250.191.74:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:47134 142.251.46.170:443 172.21.3.78:46168 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:00.772Z] "- - -" 0 - - - "-" 1101 20315 239 - "-" "-" "-" "-" "142.251.32.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:38842 142.251.46.234:443 172.21.3.78:36646 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:00.773Z] "- - -" 0 - - - "-" 1587 22306 2019 - "-" "-" "-" "-" "142.250.189.234:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:35250 142.251.46.234:443 172.21.3.78:36648 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:02.720Z] "- - -" 0 - - - "-" 1101 20315 206 - "-" "-" "-" "-" "142.251.32.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:38854 172.217.164.106:443 172.21.3.78:54774 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:11.082Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 142.251.32.42:443 172.21.3.78:48676 safebrowsingohttpgateway.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:31.246Z] "- - -" 0 - - - "-" 13182 6838963 49867 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:33878 172.20.1.241:443 172.21.3.78:33844 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:31.250Z] "- - -" 0 - - - "-" 17248 242205 62803 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:33880 172.20.1.241:443 172.21.3.78:33860 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:23.798Z] "- - -" 0 - - - "-" 2095 158687 12397 - "-" "-" "-" "-" "142.251.46.170:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:45348 172.217.164.106:443 172.21.3.78:35532 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:36.347Z] "- - -" 0 - - - "-" 1109 20315 153 - "-" "-" "-" "-" "142.250.189.202:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:39536 142.251.214.138:443 172.21.3.78:37374 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:36.341Z] "- - -" 0 - - - "-" 1092 42683 195 - "-" "-" "-" "-" "142.251.32.42:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:41540 172.217.12.106:443 172.21.3.78:34996 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:36.346Z] "- - -" 0 - - - "-" 1109 20315 202 - "-" "-" "-" "-" "142.251.46.170:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:56628 142.250.191.74:443 172.21.3.78:56148 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:00.400Z] "- - -" 0 - - - "-" 4519 57962 37525 - "-" "-" "-" "-" "142.251.46.202:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:46490 142.251.214.138:443 172.21.3.78:41582 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:36.346Z] "- - -" 0 - - - "-" 1092 20315 1607 - "-" "-" "-" "-" "142.250.191.74:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:56150 172.217.12.106:443 172.21.3.78:35012 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:38.354Z] "- - -" 0 - - - "-" 1113 20315 228 - "-" "-" "-" "-" "172.217.12.106:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:34042 142.251.32.42:443 172.21.3.78:55244 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:38.493Z] "- - -" 0 - - - "-" 1092 20315 255 - "-" "-" "-" "-" "142.251.214.138:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:59770 142.250.189.170:443 172.21.3.78:55620 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:38.544Z] "- - -" 0 - - - "-" 1092 20315 245 - "-" "-" "-" "-" "142.250.189.170:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:55626 172.217.164.106:443 172.21.3.78:52084 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:58.205Z] "- - -" 0 - - - "-" 1113 34295 1388 - "-" "-" "-" "-" "142.250.189.234:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:49932 142.251.46.234:443 172.21.3.78:34442 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:58.536Z] "- - -" 0 - - - "-" 1113 42683 1981 - "-" "-" "-" "-" "142.250.189.202:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:59338 142.251.214.138:443 172.21.3.78:48970 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:19.480Z] "- - -" 0 - - - "-" 17289 4194026 107316 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:44162 172.20.1.241:443 172.21.3.78:44150 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:31.246Z] "- - -" 0 - - - "-" 19179 352665 95551 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:33854 172.20.1.241:443 172.21.3.78:33830 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:59.147Z] "- - -" 0 - - - "-" 8851 89102 67659 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:34680 172.20.1.241:443 172.21.3.78:34666 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:08.538Z] "- - -" 0 - - - "-" 1117 20315 250 - "-" "-" "-" "-" "142.250.189.170:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:60886 142.250.189.234:443 172.21.3.78:56400 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:11.539Z] "- - -" 0 - - - "-" 1117 20315 381 - "-" "-" "-" "-" "142.250.189.234:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:56402 142.250.189.170:443 172.21.3.78:60892 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:13.100Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 142.251.214.138:443 172.21.3.78:46176 safebrowsingohttpgateway.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:22.531Z] "- - -" 0 - - - "-" 1117 20315 747 - "-" "-" "-" "-" "142.251.46.234:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:36602 142.250.189.202:443 172.21.3.78:33906 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:30.779Z] "- - -" 0 - - - "-" 1117 20315 1187 - "-" "-" "-" "-" "172.217.12.106:443" outbound|443||optimizationguide-pa.googleapis.com 172.21.3.78:56030 142.251.46.234:443 172.21.3.78:52306 optimizationguide-pa.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:07.647Z] "- - -" 0 - - - "-" 3443 1968 32433 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:39628 172.20.1.241:443 172.21.3.78:39608 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:07.665Z] "- - -" 0 - - - "-" 10279 27890 32415 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:39684 172.20.1.241:443 172.21.3.78:39670 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:17:34.554Z] "- - -" 0 - - - "-" 7616 247743 65533 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:36314 172.20.1.241:443 172.21.3.78:36298 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:40.962Z] "- - -" 0 - - - "-" 3429 2014 32443 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:50346 172.20.1.241:443 172.21.3.78:50344 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:07.653Z] "- - -" 0 - - - "-" 11833 28912 65752 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:39664 172.20.1.241:443 172.21.3.78:39652 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:07.648Z] "- - -" 0 - - - "-" 4690 59991 65765 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:39644 172.20.1.241:443 172.21.3.78:39616 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:19:27.170734Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:155 StreamSecrets gRPC config stream to sds-grpc closed: 13, thread=11 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:19:27.170770Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:155 StreamSecrets gRPC config stream to sds-grpc closed: 13, thread=11 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:19:27.170834Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:155 DeltaAggregatedResources gRPC config stream to xds-grpc closed: 13, thread=11 [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:19:27.170811Z info ads ADS: "@" loki-cypress-test.logging-2 terminated [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:19:27.170876Z info ads ADS: "@" loki-cypress-test.logging-1 terminated [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:18:07.647Z] "- - -" 0 - - - "-" 5009 3022 79403 - "-" "-" "-" "-" "172.20.1.241:443" outbound|443||grafana.dev.bigbang.mil 172.21.3.78:39622 172.20.1.241:443 172.21.3.78:39594 grafana.dev.bigbang.mil - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:59.331Z] "- - -" 0 - - - "-" 1889 148929 147725 - "-" "-" "-" "-" "34.120.177.193:443" outbound|443||grafana.com 172.21.3.78:41528 34.120.177.193:443 172.21.3.78:41508 grafana.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:59.333Z] "- - -" 0 - - - "-" 1856 767973 147723 - "-" "-" "-" "-" "34.120.177.193:443" outbound|443||grafana.com 172.21.3.78:41550 34.120.177.193:443 172.21.3.78:41516 grafana.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:59.330Z] "- - -" 0 - - - "-" 3210 364197 147725 - "-" "-" "-" "-" "34.120.177.193:443" outbound|443||grafana.com 172.21.3.78:41524 34.120.177.193:443 172.21.3.78:41498 grafana.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:59.136Z] "- - -" 0 - - - "-" 2805 642002 147921 - "-" "-" "-" "-" "34.120.177.193:443" outbound|443||grafana.com 172.21.3.78:41492 34.120.177.193:443 172.21.3.78:41476 grafana.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:59.336Z] "- - -" 0 - - - "-" 3067 680321 147721 - "-" "-" "-" "-" "34.120.177.193:443" outbound|443||grafana.com 172.21.3.78:41566 34.120.177.193:443 172.21.3.78:41518 grafana.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:59.336Z] "- - -" 0 - - - "-" 3173 409037 147721 - "-" "-" "-" "-" "34.120.177.193:443" outbound|443||grafana.com 172.21.3.78:41570 34.120.177.193:443 172.21.3.78:41534 grafana.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:35.364Z] "- - -" 0 - - - "-" 696 8205 171708 - "-" "-" "-" "-" "142.251.214.142:443" outbound|443||redirector.gvt1.com 172.21.3.78:34980 142.250.189.206:443 172.21.3.78:50724 redirector.gvt1.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:35.400Z] "- - -" 0 - - - "-" 4123 8164 171674 - "-" "-" "-" "-" "142.251.46.234:443" outbound|443||content-autofill.googleapis.com 172.21.3.78:46428 172.217.164.106:443 172.21.3.78:49800 content-autofill.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:09.057Z] "- - -" 0 - - - "-" 974 66182 198018 - "-" "-" "-" "-" "142.251.32.35:443" outbound|443||clientservices.googleapis.com 172.21.3.78:45794 142.251.32.35:443 172.21.3.78:45780 clientservices.googleapis.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:09.055Z] "- - -" 0 - - - "-" 821 6816 198035 - "-" "-" "-" "-" "142.251.2.84:443" outbound|443||accounts.google.com 172.21.3.78:45624 142.251.2.84:443 172.21.3.78:45614 accounts.google.com - traceID=- [pod/loki-cypress-test/istio-proxy] [2024-10-03T20:16:03.785Z] "- - -" 0 - - - "-" 867 286714 203378 - "-" "-" "-" "-" "104.16.24.34:443" outbound|443||registry.npmjs.org 172.21.3.78:51608 104.16.2.35:443 172.21.3.78:49302 registry.npmjs.org - traceID=- [pod/loki-cypress-test/istio-proxy] 2024-10-03T20:19:27.206442Z info Envoy exited normally -
I noticed a few points of failure, see the below timestamps to the referenced cypress/video/01-loki-health.spec.cy.js.mp4
-
The get [data-testid="data-testid Alert Success"] only waits 10 seconds (timestamp 0:29-0:39) before giving an error even though timeout: 180000 so the timeout variable isn't actually increasing the time out since 180000ms is 3 minutes.
-
After failure the Loki Test attempts a retry of cy.performGrafanaLogin('admin', 'prom-operator') the test will fail because it doesn't account that the user is already logged from the first test, so it will fail with a AssertionError: Timed out retrying after 12000ms: Expected to find element: "input[name="user"]", but never found it. (timestamp 1:10)
I think it is a matter of increasing the timeout as there is a pattern that the the loki-cypress-test always at upgrade but fails at clean_install which might make sense since the upgrade cluster has longer time for the loki endpoint to accept service.
-
mentioned in merge request !281 (merged)
-
removed statusdoing label
added statusreview label
changed iteration to Big Bang Iterations Oct 1, 2024 - Oct 14, 2024
mentioned in merge request big-bang/bigbang!5212 (merged)
closed with merge request big-bang/bigbang!5212 (merged)
mentioned in commit big-bang/bigbang@8550236d
removed statusreview label
