UNCLASSIFIED - NO CUI

Skip to content

Add wait logic to test cases.

Current logic at some of the tests for the test suite have pass/fail/inconclusive.

Preliminary work has begun to implement wait logic in the initial test framework and now needs to be added to the test cases. the current inconclusive status should be replaced with the more precise waiting status for better deployment sequence awareness.

Acceptance Criteria

Core Function Implementation

  • waiting_test() function implemented with proper exit code 2, standardized message with emoji, and consistent pattern with existing functions
  • inconclusive_test() function removed and all usage migrated to either waiting_test() or fail_test()

Test Runner Integration

  • Test runner handles exit code 2 as "waiting" status, writes correct marker files, preserves logs, and displays appropriate messages
  • Stage evaluation tracks waiting tests separately, includes in retry logic, and marks stages incomplete when waiting tests exist

API and Results Processing

  • API endpoints support waiting status in filters, /api/tests/status/waiting endpoint, and clean 3-status model in JSON responses
  • HTML display shows waiting tests with icons, clean visual model (), and clear "not ready yet" indication

Deployment Integration Testing

  • All deployment stages (4-6) handle waiting logic properly with correct retry behavior and eventual transition to pass/fail
  • System maintains backward compatibility while supporting clean pass/fail/waiting model throughout

Documentation and Error Handling

  • Documentation includes clear examples, deployment sequence patterns, and waiting vs fail guidance
  • System provides clear logging, handles edge cases gracefully, and maintains stability during waiting operations

Existing Functions in python_test_functions.py

def pass_test(test_id):     # exit(0)
def fail_test(test_id):     # exit(1) 
def inconclusive_test(test_id):  # exit(2) - TO BE DEPRECATED
# waiting_test() is MISSING!

Proposed Clean Status Model

def pass_test(test_id):     # exit(0) - Test passed
def fail_test(test_id):     # exit(1) - Test failed  
def waiting_test(test_id):  # exit(2) - Test waiting for deployment
# inconclusive_test() - REMOVE (replace with waiting where appropriate)

Test Usage Pattern

Tests are using waiting logic for deployment sequence awareness:

if deployment_not_ready:
    waiting_test(test_id)  # ← This function doesn't exist!

Required Implementation

1. Add Missing Function to python_test_functions.py

def waiting_test(test_id):
    """Standardized message and exit behavior for a test case that needs to wait."""
    print("")
    print("⏳ %s :: WAITING" % test_id)
    sys.exit(2)  # Use exit code 2 for waiting status (replacing inconclusive)

2. Remove Inconclusive Function

# REMOVE this function entirely - replace usage with waiting_test()
# def inconclusive_test(test_id):
#     sys.exit(2)

3. Update Test Runner Logic

The test runner currently handles exit code 2 as "inconclusive". Need to:

  • Handle exit code 2 as "waiting" status (replacing inconclusive)
  • Update retry logic to handle waiting tests appropriately
  • Remove all inconclusive handling and replace with waiting logic

4. Update Results Server and Rendering

  • Update results_server.py to handle "waiting" status in marker files (replacing inconclusive)
  • Update render_results.py to properly display waiting tests
  • Ensure API endpoints return waiting status correctly
  • Remove all inconclusive references and replace with waiting

Benefits of Test-Case Level Waiting Logic

  1. Precision: Tests can determine exactly when to wait vs fail
  2. Context: Tests understand their position in deployment sequence
  3. Efficiency: Avoid retrying tests that will definitely fail
  4. Clarity: Clean pass/fail/waiting model removes ambiguous "inconclusive" status

Example Use Cases

Current Working Pattern (once function is added)

# Crossplane provider not ready yet
if provider_status != "Healthy":
    waiting_test(test_id)  # Will work once function exists

# BigBang components installing
if helm_release_status == "pending-install":
    waiting_test(test_id)  # Smart waiting

# Edge configuration not applied yet  
if edge_config_objects == 0:
    waiting_test(test_id)  # Deployment sequence awareness

Related Work

This issue builds on the recently implemented waiting logic in the test runner:

  • Pass/Fail/Waiting Logic - Enhanced test status system
  • Daemon Test Execution - Parallel test running with smart retries
  • API Endpoints - REST API for test status monitoring

Implementation Notes

The waiting logic should replace inconclusive status entirely:

  • Remove inconclusive_test() function from python_test_functions.py
  • Migrate existing inconclusive usage to either waiting_test() or fail_test()
  • Use clean pass(0)/fail(1)/waiting(2) exit code model

These stages have natural dependencies where waiting logic provides the most value.

Edited by Alfredo Diaz