Run only the tests a change touches
Updated
Most runners can pick tests from a change on one machine: Jest's --onlyChanged, Vitest's --changed, pytest-testmon, and go test's cache. To leave out every test that already passed on the files it reads, wherever it passed, on a laptop, in CI or in an agent's sandbox, wrap the runner in hushrun run and set tests in hushrun.yaml.
How these approaches differ, tool by tool: Running only affected tests.
How hushrun decides
With test reuse on, hushrun records the files each test file, or each test, read when it passed. After a change, it runs the tests whose files changed and leaves out the rest:
hushrun: unit: 3 of 101 tests run, in 2 of 2 test files; 98 already passed on the files they read, and 2 more that already passed run anyway, as a checkIt's off until you turn it on, check by check. It needs hushrun 0.16.0 or later for single tests, 0.13.0 for whole test files, and results on: your organization's backend, or this machine's cache (Where results go).
Turn it on for one check
Run the tests as a named check, with the runner in the command, such as
hushrun run unit -- pytest. A script that hides the runner, such asnpm test, gets no test left out, except for pytest, CTest andswift test(Check what your runner needs).Add the check to
hushrun.yamlinobserve, so every test still runs and hushrun says what it would have left out:hushrun.yaml checks: unit: tests: mode: observe unit: testMake a few ordinary changes and read the line it prints before the runner starts:
hushrun: unit: with `tests: reuse`, 2 of 13 tests would have run, in 2 of 5 test files; 11 already passed on the files they readChange
observetoreuse, and commithushrun.yaml, so laptops, CI and sandboxes all reuse.
To ask why one test ran, use hushrun why unit tests/test_cart.py::test_one. Every setting: Reuse tests one by one.
pytest
- On its own: pytest-testmon records which code each test ran, with Coverage.py. Install it with
pip install pytest-testmonand runpytest --testmon. Its data stays in.testmondata. - With hushrun:
hushrun run unit -- pytest, orpython -m pytest. Nothing to add. Not under tox or nox, or a Python started with-Ior-E.
Vitest
- On its own:
vitest --changedruns the tests affected by changed files, andvitest related src/cart.tsthe tests that cover the files you name. Both follow static imports only. - With hushrun:
hushrun run unit -- vitest run, plus one entry in Vitest's config that loads hushrun's plugin: the line. With Vitest's coverage on,typecheckon orisolate: false, every test runs.
Jest
- On its own:
jest --onlyChangedruns the tests related to changed files in git or Mercurial,--changedSince=origin/maincompares with a branch, and--findRelatedTeststakes files, as in a pre-commit hook. They need static imports. - With hushrun:
hushrun run unit -- jest, on Jest 29 or 30. Nothing to add, unless a script replacesNODE_OPTIONS: what to do. With Jest's coverage on, every test file runs.
Go
- On its own:
go test ./...caches each package's successful result on the machine, and replays it while the test binary, the flags, the files the tests opened and the variables they read stay the same.-count=1turns it off. - With hushrun:
hushrun run unit -- go test ./.... Go 1.20 or later leaves out single tests; older Go, whole packages. A command with-runor-skip, and a package with aTestMain, reuse by package.
Gradle
For Java, Kotlin and Android's local unit tests.
- On its own: with
--build-cache, ororg.gradle.caching=trueingradle.properties, Gradle reuses aTesttask's outputs when its inputs match, and a remote cache shares them. Develocity's Predictive Test Selection, an add-on, picks the tests likely to matter from past builds. - With hushrun:
hushrun run unit -- ./gradlew test, plus one line insettings.gradleorsettings.gradle.kts: the line. Kotest's tests aren't left out one by one.
Maven
- On its own: the build cache extension keys each project on its sources, project model, plugins and their parameters, restores projects whose key matches, and can share through a remote cache.
- With hushrun:
hushrun run unit -- mvn test, or./mvnw verify. Nothing to add. JUnit 5 needs Surefire 3.0.0-M4 or later; JUnit 4 and TestNG, Surefire 3.6 or later.
swift test
- With hushrun:
hushrun run unit -- swift test, on Swift 6.0 or later, on macOS or Linux. Nothing to add. Not with--enable-code-coverageor--skip-build.
CTest
For C and C++.
- With hushrun:
hushrun run unit -- ctest --test-dir build, on CTest 3.21 or later. Build before you runctest: a test that ran code older than its sources records no pass.make testandninja testaren't watched.
Rust
cargo test and cargo nextest run reuse whole test binaries, never one test. Compiled crates can be reused too: A Rust build cache in GitHub Actions.
What to expect
- A change outside the sources runs every test. A README, a lockfile or a CI file may be read by any test. List the files no test reads in
tests.ignore. - Collecting and building still happen, so leaving out 95% of the tests saves less than 95% of the time.
- A few tests run anyway, as a check. One that fails though it passed on the same files suspends reuse for the check until a person runs
hushrun tests resume unit. - Reports hold only what ran, and coverage isn't recorded for such a run.
More in The trades and Limits.
Related
- Reuse tests one by one: the full setup.
- Running only affected tests: the tools side by side.
- Cache test results across machines
- hushrun on your machine: free, with no account.