Re-add upgrade notice parsing

General MR

Summary

Trying this again.

Readds the logic from !472 (merged). All the comments there are still valid.

Additionally, I now have a more "complete" testing script to reduce the testing period dramatically. This will run locally on a Mac and just needs to be slightly updated to include the relevant information for a BB fork and a token:

#!/usr/bin/env bash

# debug
# set -o xtrace

# Don't change
BB_AUTO_MR_TOKEN=""
MR_BOT_TOKEN="$BB_AUTO_MR_TOKEN"
BB_MR_REVIEWER_IDS=""

# Project specific
# ## eck-operator
# CI_PROJECT_ID="big-bang%2fproduct%2fpackages%2feck-operator"
# CHANGELOG_URL="https://repo1.dso.mil/big-bang/product/packages/eck-operator/-/blob/b9e0680286fd71681d8ff2d1925d7bbd746e2b49/CHANGELOG.md"
# PACKAGE_MR_URL="https://repo1.dso.mil/big-bang/product/packages/eck-operator/-/merge_requests/88/"
# export CI_COMMIT_SHA="b6356aa6932e3383b6dd6bc700d4e09abbd3c676"

## elasticsearch-kibana
CI_PROJECT_ID="big-bang%2fproduct%2fpackages%2felasticsearch-kibana"
CHANGELOG_URL="https://repo1.dso.mil/big-bang/product/packages/elasticsearch-kibana/-/blob/5dc149ac34791d5942d99d8f4c0acdd2e24218ac/CHANGELOG.md"
PACKAGE_MR_URL="https://repo1.dso.mil/big-bang/product/packages/elasticsearch-kibana/-/merge_requests/177"
export CI_COMMIT_SHA="5dc149ac34791d5942d99d8f4c0acdd2e24218ac"

## Phony MR in kiali, requires changing some other things below
# CI_PROJECT_ID="big-bang%2fproduct%2fpackages%2fkiali"
# CHANGELOG_URL="https://repo1.dso.mil/big-bang/product/packages/kiali/-/blob/ea59ca27d73d706ff86496380da0f9f4df86e15c/CHANGELOG.md"
# PACKAGE_MR_URL="https://repo1.dso.mil/big-bang/product/packages/kiali/-/merge_requests/125"
# export CI_COMMIT_SHA="5dc149ac34791d5942d99d8f4c0acdd2e24218ac"

create_bigbang_merge_request() {
  echo -e "\e[0Ksection_start:`date +%s`:create_bigbang_merge_request[collapsed=true]\r\e[0K\e[33;1mCreating Big Bang Merge Request\e[37m"
  GITLAB_PROJECTS_API_ENDPOINT="https://repo1.dso.mil/api/v4/projects"

  # Get a list of most recently merged package MRs
  curl -s "${GITLAB_PROJECTS_API_ENDPOINT}/${CI_PROJECT_ID}/merge_requests?state=merged&sort_by=updated_at" > package_mr_list.json
  # TODO use this for testing against phony MR
  # curl -s "${GITLAB_PROJECTS_API_ENDPOINT}/${CI_PROJECT_ID}/merge_requests?sort_by=updated_at" > package_mr_list.json
  # Filter to find the one with the same commit as this `main` pipeline
 
  PACKAGE_MR_ID=$(yq '.[] | select(.merge_commit_sha == strenv(CI_COMMIT_SHA)) | .iid' package_mr_list.json)
  # TODO use this for testing against phony MR
  # PACKAGE_MR_ID=125
  # If MR not found in Gitlab, exit with warnings
  # Safeguard against pushes direct to `main`, etc
  if [[ -z ${PACKAGE_MR_ID} ]]; then
    echo "Package MR not found, skipping auto Big Bang merge request."
    exit 201
  fi

  # Parse upgrade notices from description
  DESCRIPTION=$(yq '.[] | select(.merge_commit_sha == strenv(CI_COMMIT_SHA)) | .description' package_mr_list.json)
  # TODO use this for testing against phony MR
  # DESCRIPTION=$(yq '.[] | select(.iid == 125) | .description' package_mr_list.json)
  UPGRADE_NOTICES=$(get_upgrade_notices "$DESCRIPTION")

  ## Update merge request with reviewers and a description
  # Get ID of the MR that was just created
  # TODO this line will error but it's okay
  BB_MR_ID=$(curl -s "${GITLAB_PROJECTS_API_ENDPOINT}/${BB_PROJECT_ID}/merge_requests?source_branch=${BB_SOURCE_BRANCH}&state=opened" | jq '.[].iid' | head -1)
  # TODO hardcode fork MR id to test with
  BB_MR_ID="29"

  # Get description of MR and save it to a JSON file
  JSON_DESCRIPTION_FILE="/tmp/description.json"
  DESCTEMPLATES_REPO="https://repo1.dso.mil/big-bang/gitlab-description-templates"
  curl -s "${DESCTEMPLATES_REPO}/-/raw/main/.gitlab/merge_request_templates/package_owner_bb.json" > ${JSON_DESCRIPTION_FILE}

  # TODO change gsed to sed before
  # Edit the JSON file by adding curly brackets and "description" to make it a valid JSON request to the GitLab API
  gsed -i 's|^|\{\"description\"\:|' ${JSON_DESCRIPTION_FILE}
  gsed -i 's|$|\}|' ${JSON_DESCRIPTION_FILE}

  # Update description JSON file with package changes
  gsed -i "s|(Describe Package changes here)|${CHANGELOG_URL}|g" ${JSON_DESCRIPTION_FILE}

  # Update description of MR with the package MR URL
  gsed -i "s|(Link to Package MR here)|${PACKAGE_MR_URL}|g" ${JSON_DESCRIPTION_FILE}

  # Update description of MR with the upgrade notices
  # Replace the default boilerplate with the parsed upgrade notices
  json=$(cat ${JSON_DESCRIPTION_FILE})
  replaced=${json//\(Include any relevant notes about upgrades here or write \\\"N\/A\\\" if there are none\)/${UPGRADE_NOTICES}}
  # Update the saved description body
  echo ${replaced} > ${JSON_DESCRIPTION_FILE}

  # TODO hardcode fork project id
  BB_PROJECT_ID="14950"

  # Update description of MR with package changes from CHANGELOG.md and add reviewers
  curl -s --request PUT --header "Content-Type: application/json" --header "PRIVATE-TOKEN: ${BB_AUTO_MR_TOKEN}" --data "@${JSON_DESCRIPTION_FILE}" "${GITLAB_PROJECTS_API_ENDPOINT}/${BB_PROJECT_ID}/merge_requests/${BB_MR_ID}?reviewer_ids=${BB_MR_REVIEWER_IDS}"

  # MR Link
  echo "✅ Big Bang MR created: https://${BB_REPO}/-/merge_requests/${BB_MR_ID}"

  echo -e "\e[0Ksection_end:`date +%s`:create_bigbang_merge_request\r\e[0K"
}


validate_bigbang_merge_request() {
  echo -e "\e[0Ksection_start:`date +%s`:validate_bigbang_merge_request[collapsed=true]\r\e[0K\e[33;1mValidating Big Bang Merge Request\e[37m"

  # Replace these variables with your GitLab details
  GITLAB_API_URL="https://repo1.dso.mil/api/v4"
  PRIVATE_TOKEN="$MR_BOT_TOKEN"
  PROJECT_ID="$CI_PROJECT_ID"
  MERGE_REQUEST_IID="$CI_MERGE_REQUEST_IID"

  # Enable these variables and point them at the correct project / MR ID to test
  # PROJECT_ID=""
  # MERGE_REQUEST_IID=""

  url="$GITLAB_API_URL/projects/$PROJECT_ID/merge_requests/$MERGE_REQUEST_IID"

  description=$(fetch_mr_description "$url" 2 "$PRIVATE_TOKEN")
  upgrade_notices=$(get_upgrade_notices "$description")

  if [ $? -ne 0 ]; then
    echo "No Upgrade Notices section found in the description."
    echo "Please include an ## Upgrade Notices section and list upgrade notes beneath it."
    exit 1
  fi
  
  # Check the value of upgrade_notices
  if [[ "$upgrade_notices" == "(Include any relevant notes about upgrades here or write \"N/A\" if there are none)" ]]; then
    echo "Upgrade notices were not filled out. Please include upgrade notices or write \"N/A\" if there are none."
    exit 1
  elif [[ "$upgrade_notices" == "N/A" || "$upgrade_notices" == "n/a" || "$upgrade_notices" == "na" || "$upgrade_notices" == "None" || -n "$upgrade_notices" ]]; then
    # Print the result
    echo "✅ Upgrade Notices: $upgrade_notices"
  else
    # If upgrade_notices is not acceptable, fail the CI/CD pipeline
    echo "Invalid value for Upgrade Notices: $upgrade_notices"
    exit 1
  fi

  echo -e "\e[0Ksection_end:`date +%s`:validate_bigbang_merge_request\r\e[0K"
}

# Fetches a merge request with retry-logic and returns the description
fetch_mr_description() {
  local url="$1"
  local max_attempts="$2"
  local token="$3"

  # Query GitLab API to get merge request details
  # Try the request with the token first, retrying without the token if it fails
  # The token is only valid for 1 year but only explicitly required for non-public repos
  # of which BigBang and the package repos are currently not
  for ((attempt=1; attempt<=max_attempts; attempt++)); do
    echo "Requesting $url" >&2
    if [ $attempt -eq 1 ]; then
      response=$(curl -s -w "%{http_code}" --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "$url") 
    else
      response=$(curl -s -w "%{http_code}" "$url")
    fi

    http_status="${response: -3}"
    data="${response:0:${#response}-3}"

    if [ "$http_status" == "200" ]; then
      break
    fi

    if [ $attempt -eq $max_attempts ]; then
      echo "Failed to get merge request data"
      return 1
    fi
  done

  description=$(echo "$data" | jq -r '.description')

  if [ -z "$description" ]; then
    echo "Error: Description is empty" >&2
    return 1
  fi

  echo "$description"
}

# Fetch the upgrade notices at $1 (the url) and return them. R
# Parses out the upgrade notices from the description body passed as the first arg
get_upgrade_notices() {
  local description="$1"

  # Drop first and last characters (extra quotes that will mess up the insertion)
  description=${description:1:-1}

  if [[ "${description}" =~ (## Upgrade Notices(.*))(## |$) ]]; then

    # Take the second capture group, the content after the header itself
    upgrade_notices=${BASH_REMATCH[2]}
    echo "ℹ️ Found $upgrade_notices" >&2
    # Trim leading whitespace, stop at <!-- comment string
    upgrade_notices=${upgrade_notices#^[[:space:]]*}
    upgrade_notices=${upgrade_notices%%<!--*}
    
    # Return
    echo "ℹ️ Returning ${upgrade_notices}" >&2 
    echo ${upgrade_notices//-/}
  else
    echo "No upgrade notices found" >&2
    return 1
  fi
}

create_bigbang_merge_request
# validate_bigbang_merge_request

Relevant logs/screenshots

Running it against the linked elasticsearch-kibana MR yields an MR description that looks like:

Linked Issue

#296 (closed)

Upgrade Notices

N/A