fix

CategoryGeneral
AuthorAlireza Rezvani
LicenseMIT
Rating4.70/5
Uses13.4K

Fix Failing or Flaky Tests

Diagnose and fix a Playwright test that fails or passes intermittently using a systematic taxonomy.

Input

$ARGUMENTS contains:

  • A test file path: e2e/login.spec.ts

  • A test name: ""should redirect after login"

  • A description: "the checkout test fails in CI but passes locally"

Steps

1. Reproduce the Failure

Run the test to capture the error:

bash
npx playwright test <file> --reporter=list

If the test passes, it's likely flaky. Run burn-in:

bash
npx playwright test <file> --repeat-each=10 --reporter=list

If it still passes, try with parallel workers:

bash
npx playwright test --fully-parallel --workers=4 --repeat-each=5

2. Capture Trace

Run with full tracing:

bash
npx playwright test <file> --trace=on --retries=0

Read the trace output. Use /debug to analyze trace files if available.

3. Categorize the Failure

Load flaky-taxonomy.md from this skill directory.

Every failing test falls into one of four categories:

| Category | Symptom | Diagnosis |
|---|---|---|
| Timing/Async | Fails intermittently everywhere | --repeat-each=20 reproduces locally |
| Test Isolation | Fails in suite, passes alone | --workers=1 --grep "test name" passes |
| Environment | Fails in CI, passes locally | Compare CI vs local screenshots/traces |
| Infrastructure | Random, no pattern | Error references browser internals |

4. Apply Targeted Fix

Timing/Async:

  • Replace waitForTimeout() with web-first assertions

  • Add await to missing Playwright calls

  • Wait for specific network responses before asserting

  • Use toBeVisible() before interacting with elements

Test Isolation:

  • Remove shared mutable state between tests

  • Create test data per-test via API or fixtures

  • Use unique identifiers (timestamps, random strings) for test data

  • Check for database state leaks

Environment:

  • Match viewport sizes between local and CI

  • Account for font rendering differences in screenshots

  • Use docker locally to match CI environment

  • Check for timezone-dependent assertions

Infrastructure:

  • Increase timeout for slow CI runners

  • Add retries in CI config (retries: 2)

  • Check for browser OOM (reduce parallel workers)

  • Ensure browser dependencies are installed

5. Verify the Fix

Run the test 10 times to confirm stability:

bash
npx playwright test <file> --repeat-each=10 --reporter=list

All 10 must pass. If any fail, go back to step 3.

6. Prevent Recurrence

Suggest:

  • Add to CI with retries: 2 if not already

  • Enable trace: 'on-first-retry'` in config

  • Add the fix pattern to project's test conventions doc

Output

  • Root cause category and specific issue
  • The fix applied (with diff)
  • Verification result (10/10 passes)
  • Prevention recommendation
Join our Telegram