Add wait logic to test cases.
Current logic at some of the tests for the test suite have pass/fail/inconclusive.
Preliminary work has begun to implement wait logic in the initial test framework and now needs to be added to the test cases. the current inconclusive status should be replaced with the more precise waiting status for better deployment sequence awareness.
Acceptance Criteria
Core Function Implementation
-
waiting_test()
function implemented with proper exit code 2, standardized message with⏳ emoji, and consistent pattern with existing functions -
inconclusive_test()
function removed and all usage migrated to eitherwaiting_test()
orfail_test()
Test Runner Integration
-
Test runner handles exit code 2 as "waiting" status, writes correct marker files, preserves logs, and displays appropriate messages -
Stage evaluation tracks waiting tests separately, includes in retry logic, and marks stages incomplete when waiting tests exist
API and Results Processing
-
API endpoints support waiting status in filters, /api/tests/status/waiting
endpoint, and clean 3-status model in JSON responses -
HTML display shows waiting tests with ⏳ icons, clean visual model (✅ ❌ ⏳ ), and clear "not ready yet" indication
Deployment Integration Testing
-
All deployment stages (4-6) handle waiting logic properly with correct retry behavior and eventual transition to pass/fail -
System maintains backward compatibility while supporting clean pass/fail/waiting model throughout
Documentation and Error Handling
-
Documentation includes clear examples, deployment sequence patterns, and waiting vs fail guidance -
System provides clear logging, handles edge cases gracefully, and maintains stability during waiting operations
python_test_functions.py
Existing Functions in def pass_test(test_id): # exit(0)
def fail_test(test_id): # exit(1)
def inconclusive_test(test_id): # exit(2) - TO BE DEPRECATED
# waiting_test() is MISSING!
Proposed Clean Status Model
def pass_test(test_id): # exit(0) - Test passed
def fail_test(test_id): # exit(1) - Test failed
def waiting_test(test_id): # exit(2) - Test waiting for deployment
# inconclusive_test() - REMOVE (replace with waiting where appropriate)
Test Usage Pattern
Tests are using waiting logic for deployment sequence awareness:
if deployment_not_ready:
waiting_test(test_id) # ← This function doesn't exist!
Required Implementation
python_test_functions.py
1. Add Missing Function to def waiting_test(test_id):
"""Standardized message and exit behavior for a test case that needs to wait."""
print("")
print("⏳ %s :: WAITING" % test_id)
sys.exit(2) # Use exit code 2 for waiting status (replacing inconclusive)
2. Remove Inconclusive Function
# REMOVE this function entirely - replace usage with waiting_test()
# def inconclusive_test(test_id):
# sys.exit(2)
3. Update Test Runner Logic
The test runner currently handles exit code 2 as "inconclusive". Need to:
- Handle exit code 2 as "waiting" status (replacing inconclusive)
- Update retry logic to handle waiting tests appropriately
- Remove all inconclusive handling and replace with waiting logic
4. Update Results Server and Rendering
- Update
results_server.py
to handle "waiting" status in marker files (replacing inconclusive) - Update
render_results.py
to properly display waiting tests - Ensure API endpoints return waiting status correctly
- Remove all inconclusive references and replace with waiting
Benefits of Test-Case Level Waiting Logic
- Precision: Tests can determine exactly when to wait vs fail
- Context: Tests understand their position in deployment sequence
- Efficiency: Avoid retrying tests that will definitely fail
- Clarity: Clean pass/fail/waiting model removes ambiguous "inconclusive" status
Example Use Cases
Current Working Pattern (once function is added)
# Crossplane provider not ready yet
if provider_status != "Healthy":
waiting_test(test_id) # Will work once function exists
# BigBang components installing
if helm_release_status == "pending-install":
waiting_test(test_id) # Smart waiting
# Edge configuration not applied yet
if edge_config_objects == 0:
waiting_test(test_id) # Deployment sequence awareness
Related Work
This issue builds on the recently implemented waiting logic in the test runner:
- Pass/Fail/Waiting Logic - Enhanced test status system
- Daemon Test Execution - Parallel test running with smart retries
- API Endpoints - REST API for test status monitoring
Implementation Notes
The waiting logic should replace inconclusive status entirely:
- Remove
inconclusive_test()
function frompython_test_functions.py
- Migrate existing inconclusive usage to either
waiting_test()
orfail_test()
- Use clean pass(0)/fail(1)/waiting(2) exit code model
These stages have natural dependencies where waiting logic provides the most value.