`vault` blocks BB upgrade in all-packges pipeline

Issue

When an all-packages pipeline runs in BB umbrella, vault does not upgrade cleanly.

Example MR with failing upgrade pipeline: big-bang/bigbang!5723

Relevant logs:

timed out waiting for the condition on jobs/vault-vault-job-init

Acceptance Criteria

• No vault components block the BB upgrade in all-packages pipelines

added kindci priority5 statusready-to-work teamStorage & Collaboration vault labels

set weight to 2

mentioned in merge request big-bang/bigbang!5718 (merged)

marked this issue as blocking kyverno-policies#150 (closed)

changed iteration to Big Bang Iterations Feb 18, 2025 - Mar 3, 2025

changed milestone to %2.48.0

assigned to @pchristiel

changed iteration to Big Bang Iterations Mar 4, 2025 - Mar 17, 2025

added statusdoing label and removed statusready-to-work label

I ran this command to check the vault logs in real time;

kubectl logs -f vault-vault-0 -n vault -c vault

and I found that:

Vault nodes cannot resolve the DNS names of other Vault nodes in the HA cluster.

and this issue prevent vault to upgrading properly.

Summary:

The inability of the Vault nodes to find each other via DNS is a fundamental barrier to a successful HA Vault deployment and upgrade. The upgrade process relies on the nodes coordinating, electing a leader, and synchronizing data – all of which are impossible without proper network communication and DNS resolution.

logs output:

y":1,"term":10,"votesNeeded":2} {"@level":"warn","@message":"Election timeout reached, restarting election","@module":"storage.raft","@timestamp":"2025-03-05T12:09:42.145931Z"} {"@level":"info","@message":"entering candidate state","@module":"storage.raft","@timestamp":"2025-03-05T12:09:42.145972Z","node":{},"term":10} {"@level":"error","@message":"failed to make requestVote RPC","@module":"storage.raft","@timestamp":"2025-03-05T12:09:42.150886Z","error":"dial tcp: lookup vault-vault-1.vault-vault-internal on 10.100.0.10:53: no such host","target":{"Suffrage":0,"ID":"vault-vault-1","Address":"vault-vault-1.vault-vault-internal:8201"},"term":10} {"@level":"error","@message":"failed to make requestVote RPC","@module":"storage.raft","@timestamp":"2025-03-05T12:09:42.151569Z","error":"dial tcp: lookup vault-vault-2.vault-vault-internal on 10.100.0.10:53: no such host","target":{"Suffrage":0,"ID":"vault-vault-2","Address":"vault-vault-2.vault-vault-internal:8201"},"term":10} {"@level":"info","@message":"pre-vote campaign failed, waiting for election timeout","@module":"storage.raft","@timestamp":"2025-03-05T12:09:42.151624Z","refused":2,"tally":1,"term":10,"votesNeeded":2} {"@level":"warn","@message":"Election timeout reached, restarting election","@module":"storage.raft","@timestamp":"2025-03-05T12:09:50.630596Z"} {"@level":"info","@message":"entering candidate state","@module":"storage.raft","@timestamp":"2025-03-05T12:09:50.630631Z","node":{},"term":10} {"@level":"error","@message":"failed to make requestVote RPC","@module":"storage.raft","@timestamp":"2025-03-05T12:09:50.634650Z","error":"dial tcp: lookup vault-vault-2.vault-vault-internal on 10.100.0.10:53: no such host","target":{"Suffrage":0,"ID":"vault-vault-2","Address":"vault-vault-2.vault-vault-internal:8201"},"term":10} {"@level":"error","@message":"failed to make requestVote RPC","@module":"storage.raft","@timestamp":"2025-03-05T12:09:50.635530Z","error":"dial tcp: lookup vault-vault-1.vault-vault-internal on 10.100.0.10:53: no such host","target":{"Suffrage":0,"ID":"vault-vault-1","Address":"vault-vault-1.vault-vault-internal:8201"},"term":10} {"@level":"info","@message":"pre-vote campaign failed, waiting for election timeout","@module":"storage.raft","@timestamp":"2025-03-05T12:09:50.635576Z","refused":2,"tally":1,"term":10,"votesNeeded":2}.

Hi @andrewshoell please can you take a look on my research comment, we will need a follow up ticket to address that.

mentioned in issue #138

marked this issue as related to #138

removed the relation with #138

marked this issue as blocked by #138

changed milestone to %2.49.0

removed iteration Big Bang Iterations Mar 4, 2025 - Mar 17, 2025

changed iteration to Big Bang Iterations Mar 18, 2025 - Mar 31, 2025