Test Flakiness Detection

Note: A flaky test is one that produces different results with the same code and test inputs. This guide explains how to integrate Cased’s flakiness detection into your workflow to identify flaky tests across your CI runs over time.

​

Understanding Flakiness Detection

​

How It Works

Cased analyzes test results across your CI pipeline runs to identify patterns of inconsistent behavior. For each test, we calculate:

​

Key Metrics

• Flakiness Score: Percentage of CI runs where the test result differed from the majority outcome (0-100%)
• Detection Pattern: How the flakiness manifests (e.g., random, timing-dependent)
• First/Last Detected: Timestamps of when flaky behavior was first and most recently observed

​

Detection Patterns

​

Common Flakiness Patterns

• Random: Results appear to be randomly inconsistent across different CI runs
• Timing: Test failures correlate with execution timing in certain CI environments
• Resource: Failures relate to system resource availability on different CI runners
• Order: Test result depends on execution order within the test suite
• Setup: Failures occur during test setup/teardown in specific CI environments

​

Flakiness Score

The flakiness score indicates how frequently a test produces inconsistent results across CI runs:

1. High Flakiness (75-100%): Test results are highly unreliable across CI runs
2. Medium Flakiness (40-74%): Test has significant stability issues in CI
3. Low Flakiness (1-39%): Test occasionally produces inconsistent results
4. Not Flaky (0%): Test produces consistent results

​

Language Integrations

​

Python

​

pytest Integration

Cased works out of the box with pytest. Configure the JSON reporter in your CI:

pip install pytest pytest-json-report
pytest --json-report --json-report-file="test-report.json"

Warning: Make sure the JSON report captures all test metadata needed for analysis.

​

Recommended pytest Configuration

​

JavaScript

​

Jest Integration

Jest provides built-in JSON reporter support. Add this to your package.json:

{
  "jest": {
    "reporters": ["default", "jest-json-reporter"]
  }
}

Install required dependency:

npm install --save-dev jest-json-reporter

Tip: Using Jest’s built-in JSON reporter eliminates the need for additional reporter plugins.

​

Ruby

​

RSpec Integration

Add the JSON formatter to your .rspec file:

--format json
--out rspec.json

​

Rust

​

Cargo Test Integration

Rust’s built-in test framework can output JSON results:

RUST_TEST_THREADS=1 cargo test --no-fail-fast -- -Z unstable-options --format json > "test-report.json"

​

Go

​

Go Test Integration

Go’s testing package supports JSON output through the -json flag:

go test -json ./... > test-report.json

For more detailed test output, you can use gotestsum:

go install gotest.tools/gotestsum@latest
gotestsum --format testname --junitfile test-report.xml --jsonfile test-report.json

Tip: gotestsum provides richer test metadata and is recommended for better flakiness detection.

​

API Reference

​

Request Format

To send test results to Cased, make a POST request to our CI agent endpoint:

https://app.cased.com/api/v1/projects/{repository}/ci-agent/

Use the following format:

{
  "repository": "owner/repo",
  "branch": "main",
  "commit_sha": "abc123...",
  "workflow_run_id": "12345",
  "runner": {
    "os": "ubuntu-latest",
    "name": "GitHub Actions",
    "cpu_count": 2,
    "memory_mb": 7168
  },
  "environment": {
    "CI": "true"
  },
  "test_results": {
    "tests": []
  }
}

Authorization:

Authorization: Bearer YOUR_CASED_TOKEN

​

CI Integration

​

GitHub Actions

Info: Here’s an example of how to integrate Cased with GitHub Actions to collect test results.

name: Test Results Collection

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  run-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Python
        run: |
          python -m pip install --upgrade pip
          pip install pytest pytest-json-report

      - name: Run tests
        run: |
          pytest --json-report --json-report-file="test-report.json"

      - name: Send results to Cased
        env:
          CASED_TOKEN: ${{ secrets.CASED_TOKEN }}
        run: |
          curl -X POST "${CASED_API_URL}/api/v1/projects/${{ github.repository }}/ci-agent/" \
            -H "Authorization: Bearer ${{ secrets.CASED_TOKEN }}" \
            -H "Content-Type: application/json" \
            -d @test-report.json

​

Best Practices

Recommended Practices:

• Consistent Environments: Use consistent CI environments to reduce false positives from environment differences
• Complete Metadata: Include detailed test and environment information in your reports
• Regular Collection: Configure test result collection for all CI runs to build comprehensive data
• Monitor Trends: Watch for patterns in flakiness across different branches and environments

​

Troubleshooting

Warning: Common issues and solutions:

• Missing Data: Ensure all required fields are included in your JSON payload
• Authentication Errors: Verify your Cased API token is correctly configured
• Incomplete Results: Check that your test reporter is configured to capture all necessary metadata%