Skip to main content

CI Test Isolation Fix: Execution Timeout Test

Problem​

The test test_execution_timeout_handling was failing in CI with a 400 "Path traversal not allowed" error instead of the expected 408 timeout error. This is a classic test isolation issue where tests pass locally but fail in CI due to different execution environments.

Root Cause​

  1. The test mocks run_graph to raise TimeoutError
  2. But path validation in _resolve_workflow_path happens before the mock is applied
  3. In CI, the CSV file setup fails or there's a race condition
  4. The validation fails before the timeout mock can be triggered

Solution Applied​

1. Enhanced Test Setup Validation​

def setUp(self):
    # Create CSV file
    self.execution_csv_path = self.create_test_csv_file(...)
    
    # Critical validation for CI
    self.assertTrue(self.execution_csv_path.exists())
    self.assertGreater(self.execution_csv_path.stat().st_size, 0)
    
    # Verify content is correct
    csv_content = self.execution_csv_path.read_text(encoding='utf-8')
    self.assertIn('testgraph', csv_content)

2. Mock Path Resolution to Bypass Validation​

def test_execution_timeout_handling(self):
    # Ensure file exists before test
    self.assertTrue(self.execution_csv_path.exists())
    
    # Mock BOTH the timeout AND path resolution
    with patch.object(self.container.graph_runner_service(), 'run_graph') as mock_run, \
         patch('agentmap.infrastructure.api.fastapi.routes.execution._resolve_workflow_path') as mock_resolve_path:
        
        mock_run.side_effect = TimeoutError("Execution timeout")
        mock_resolve_path.return_value = self.execution_csv_path  # Bypass file checks
        
        response = self.client.post("/execution/testworkflow/testgraph", json=request_data)
    
    self.assert_response_error(response, 408)

3. Created Test Isolation Utilities​

  • tests/utils/test_isolation_helpers.py provides reusable utilities
  • ensure_file_exists() for robust file validation
  • @ci_robust_test() decorator for automatic CI fixes
  • CITestValidator for comprehensive setup validation

Prevention Strategy​

For New Tests​

  1. Use the @ci_robust_test() decorator for API tests:
@ci_robust_test()
def test_api_scenario(self):
    # Test automatically made CI-robust
  1. Always validate test setup in setUp():
def setUp(self):
    super().setUp()
    # Create test files...
    
    # Validate setup
    ensure_file_exists(self.test_csv_path, "Test CSV")
  1. Mock path resolution for endpoint tests that depend on file existence:
with patch('agentmap.infrastructure.api.fastapi.routes.execution._resolve_workflow_path') as mock_resolve:
    mock_resolve.return_value = self.csv_path
    # Now test the actual business logic

General Best Practices​

  1. Always validate file existence before tests that depend on files
  2. Mock path resolution for API tests to isolate business logic from file system
  3. Use explicit file content validation to catch empty or corrupted files
  4. Test with pytest-randomly locally to catch order dependencies
  5. Use the new isolation utilities for consistent CI robustness

Why This Happens in CI​

  • Different file system behavior (Linux vs Windows)
  • Different test execution order
  • Race conditions in parallel test runs
  • Stricter security validation in CI environments
  • Different temporary directory handling

Testing the Fix​

# Run the specific test
pytest tests/fresh_suite/integration/api/test_execution_endpoints.py::TestExecutionEndpoints::test_execution_timeout_handling -v

# Run all execution tests
pytest tests/fresh_suite/integration/api/test_execution_endpoints.py -v

# Test with random order to catch isolation issues
pytest --random-order tests/fresh_suite/integration/api/test_execution_endpoints.py

This fix ensures the test behaves consistently across all environments while maintaining proper test isolation and preventing similar issues in the future.