Skip to content

Run only the tests a change touches

Updated

Most runners can pick tests from a change on one machine: Jest's --onlyChanged, Vitest's --changed, pytest-testmon, and go test's cache. To leave out every test that already passed on the files it reads, wherever it passed, on a laptop, in CI or in an agent's sandbox, wrap the runner in hushrun run and set tests in hushrun.yaml.

How these approaches differ, tool by tool: Running only affected tests.

How hushrun decides

With test reuse on, hushrun records the files each test file, or each test, read when it passed. After a change, it runs the tests whose files changed and leaves out the rest:

hushrun: unit: 3 of 101 tests run, in 2 of 2 test files; 98 already passed on the files they read, and 2 more that already passed run anyway, as a check

It's off until you turn it on, check by check. It needs hushrun 0.16.0 or later for single tests, 0.13.0 for whole test files, and results on: your organization's backend, or this machine's cache (Where results go).

Turn it on for one check

  1. Run the tests as a named check, with the runner in the command, such as hushrun run unit -- pytest. A script that hides the runner, such as npm test, gets no test left out, except for pytest, CTest and swift test (Check what your runner needs).

  2. Add the check to hushrun.yaml in observe, so every test still runs and hushrun says what it would have left out:

    hushrun.yaml
    checks:
      unit:
        tests:
          mode: observe
          unit: test
  3. Make a few ordinary changes and read the line it prints before the runner starts:

    hushrun: unit: with `tests: reuse`, 2 of 13 tests would have run, in 2 of 5 test files; 11 already passed on the files they read
  4. Change observe to reuse, and commit hushrun.yaml, so laptops, CI and sandboxes all reuse.

To ask why one test ran, use hushrun why unit tests/test_cart.py::test_one. Every setting: Reuse tests one by one.

pytest

  • On its own: pytest-testmon records which code each test ran, with Coverage.py. Install it with pip install pytest-testmon and run pytest --testmon. Its data stays in .testmondata.
  • With hushrun: hushrun run unit -- pytest, or python -m pytest. Nothing to add. Not under tox or nox, or a Python started with -I or -E.

Vitest

  • On its own: vitest --changed runs the tests affected by changed files, and vitest related src/cart.ts the tests that cover the files you name. Both follow static imports only.
  • With hushrun: hushrun run unit -- vitest run, plus one entry in Vitest's config that loads hushrun's plugin: the line. With Vitest's coverage on, typecheck on or isolate: false, every test runs.

Jest

  • On its own: jest --onlyChanged runs the tests related to changed files in git or Mercurial, --changedSince=origin/main compares with a branch, and --findRelatedTests takes files, as in a pre-commit hook. They need static imports.
  • With hushrun: hushrun run unit -- jest, on Jest 29 or 30. Nothing to add, unless a script replaces NODE_OPTIONS: what to do. With Jest's coverage on, every test file runs.

Go

  • On its own: go test ./... caches each package's successful result on the machine, and replays it while the test binary, the flags, the files the tests opened and the variables they read stay the same. -count=1 turns it off.
  • With hushrun: hushrun run unit -- go test ./.... Go 1.20 or later leaves out single tests; older Go, whole packages. A command with -run or -skip, and a package with a TestMain, reuse by package.

Gradle

For Java, Kotlin and Android's local unit tests.

  • On its own: with --build-cache, or org.gradle.caching=true in gradle.properties, Gradle reuses a Test task's outputs when its inputs match, and a remote cache shares them. Develocity's Predictive Test Selection, an add-on, picks the tests likely to matter from past builds.
  • With hushrun: hushrun run unit -- ./gradlew test, plus one line in settings.gradle or settings.gradle.kts: the line. Kotest's tests aren't left out one by one.

Maven

  • On its own: the build cache extension keys each project on its sources, project model, plugins and their parameters, restores projects whose key matches, and can share through a remote cache.
  • With hushrun: hushrun run unit -- mvn test, or ./mvnw verify. Nothing to add. JUnit 5 needs Surefire 3.0.0-M4 or later; JUnit 4 and TestNG, Surefire 3.6 or later.

swift test

  • With hushrun: hushrun run unit -- swift test, on Swift 6.0 or later, on macOS or Linux. Nothing to add. Not with --enable-code-coverage or --skip-build.

CTest

For C and C++.

  • With hushrun: hushrun run unit -- ctest --test-dir build, on CTest 3.21 or later. Build before you run ctest: a test that ran code older than its sources records no pass. make test and ninja test aren't watched.

Rust

cargo test and cargo nextest run reuse whole test binaries, never one test. Compiled crates can be reused too: A Rust build cache in GitHub Actions.

What to expect

  • A change outside the sources runs every test. A README, a lockfile or a CI file may be read by any test. List the files no test reads in tests.ignore.
  • Collecting and building still happen, so leaving out 95% of the tests saves less than 95% of the time.
  • A few tests run anyway, as a check. One that fails though it passed on the same files suspends reuse for the check until a person runs hushrun tests resume unit.
  • Reports hold only what ran, and coverage isn't recorded for such a run.

More in The trades and Limits.

Markdown