Flaky Tests

Identify and fix flaky tests that slow down your CI pipeline

Cased analyzes test results across your CI pipeline runs to identify inconsistent behavior.

Flaky tests are often caused by random or timing-dependent behavior, making them unreliable and time-consuming to debug. Cased helps you identify and fix these issues by tracking them as they occur.

For Cased to receive test results, you’ll configure your CI pipeline to send test results to Cased.

Language integrations

Python (pytest)

Cased works out of the box with pytest. Configure the JSON reporter in your CI:

pip install pytest pytest-json-report
pytest --json-report --json-report-file="test-report.json"

JavaScript (Jest)

Jest provides built-in JSON reporter support. Add this to your package.json:

{
  "jest": {
    "reporters": ["default", "jest-json-reporter"]
  }
}

Install required dependency:

npm install --save-dev jest-json-reporter

Ruby (RSpec)

Add the JSON formatter to your .rspec file:

--format json
--out rspec.json

Rust

RUST_TEST_THREADS=1 cargo test --no-fail-fast -- -Z unstable-options --format json > "test-report.json"

Go

go test -json ./... > test-report.json

or with gotestsum recommended :

go install gotest.tools/gotestsum@latest
gotestsum --format testname --junitfile test-report.xml --jsonfile test-report.json

API Reference

You can send test results to Cased in two ways:

Structured JSON

To send test results to Cased with a structured JSON payload, make a POST request to our CI agent endpoint:

https://app.cased.com/api/v1/projects/{repository}/ci-agent/

Use the following JSON format:

{
  "repository": "owner/repo",
  "branch": "main",
  "commit_sha": "abc123...",
  "workflow_run_id": "12345",
  "runner": {
    "os": "ubuntu-latest",
    "name": "GitHub Actions",
    "cpu_count": 2,
    "memory_mb": 7168
  },
  "environment": {
    "CI": "true"
  },
  "test_results": {
    "tests": []
  }
}

Include your authentication token in the header:

Authorization: Bearer YOUR_CASED_TOKEN

Raw Test Report (Python)

For Python projects, you can also send the raw pytest JSON report file:

# Generate the test report
pytest --json-report --json-report-file="test-report.json"

# Send the report to Cased
curl -X POST "https://app.cased.com/api/v1/projects/{project-name}/ci-agent/" \
  -H "Authorization: Bearer YOUR_CASED_TOKEN" \
  -H "Content-Type: application/json" \
  -d @test-report.json

where {project-name} is the name of your Cased project.

For Python projects using Poetry and Django, you might use:

# Run tests with Poetry and generate JSON report
poetry run pytest tests/ --ignore=tests/path/to/ignore/ --json-report --json-report-file="test-report.json"

# Send the report to Cased
curl -X POST "https://app.cased.com/api/v1/projects/{project-name}/ci-agent/" \
  -H "Authorization: Bearer YOUR_CASED_TOKEN" \
  -H "Content-Type: application/json" \
  -d @test-report.json

This approach works for any testing framework that generates a compatible JSON report.

CI Integration

GitHub Actions

Here’s an example of how to integrate Cased with GitHub Actions to collect test results:

name: Test Results Collection

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  run-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Python
        run: |
          python -m pip install --upgrade pip
          pip install pytest pytest-json-report

      - name: Run tests
        run: |
          pytest --json-report --json-report-file="test-report.json"

      - name: Send results to Cased
        env:
          CASED_TOKEN: ${{ secrets.CASED_TOKEN }}
        run: |
          curl -X POST "${CASED_API_URL}/api/v1/projects/${{ github.repository }}/ci-agent/" \
            -H "Authorization: Bearer ${{ secrets.CASED_TOKEN }}" \
            -H "Content-Type: application/json" \
            -d @test-report.json

For Python projects using Poetry and Django, your GitHub Actions workflow might look like this:

# ... other workflow configuration ...

steps:
  # ... checkout and setup steps ...

  - name: Run Tests
    run: |
      poetry run pytest tests/ --ignore=tests/comet/core/sources/ --json-report --json-report-file="test-report.json"
    env:
      DJANGO_SETTINGS_MODULE: comet.settings.test

  - name: Send results to Cased
    env:
      CASED_TOKEN: ${{ secrets.CASED_TOKEN }}
    run: |
      curl -X POST "https://app.cased.com/api/v1/projects/${{ github.repository }}/ci-agent/" \
        -H "Authorization: Bearer ${{ secrets.CASED_TOKEN }}" \
        -H "Content-Type: application/json" \
        -d @test-report.json

Best practices

1. Consistent environments: Use consistent CI environments
2. Complete metadata: Include test and environment information
3. Regular collection: Configure collection for all CI runs
4. Monitor trends: Watch patterns across branches and environments

Troubleshooting

Common issues:

• Missing data: Ensure required fields are in your JSON payload
• Authentication errors: Verify your Cased API token
• Incomplete results: Check that your test reporter captures all metadata