Reuse tests one by one

When a check's code changed, hushrun run normally runs every test again. With tests set in hushrun.yaml, it leaves out the tests that already passed on the files they read, and runs the rest. It's off until you turn it on, check by check.

hushrun: unit: 3 of 101 tests run, in 2 of 2 test files; 98 already passed on the files they read, and 2 more that already passed run anyway, as a check

It leaves out whole test files, or each test on its own. It starts after a miss: when the check's result key has a pass, the check skips as before, and no test runs.

Before you begin

• hushrun 0.16.0 or later, for tests one by one. Before it, hushrun reuses only whole test files.
• A check run as hushrun run NAME -- COMMAND, where COMMAND starts one of the test runners below. The check's settings go under its NAME in hushrun.yaml, which npx hushrun init writes.
• Results on: an organization's backend, or this machine's cache (Where results go). With results off, hushrun looks at no test.

Check what your runner needs

Name the runner in COMMAND, as below. A script that hides it, such as npm test or make test, gets no test left out, except for pytest, found wherever it runs, and CTest and swift test, found in a shell line, a script or a Makefile that starts them by name.

hushrun init proposes the lines for Vitest, Jest and Gradle in its list of changes to make by hand. It never writes them. Without HUSHRUN_VITEST, HUSHRUN_JEST or HUSHRUN_GRADLE, which hushrun sets for its own runs, each line does nothing.

Vitest loads hushrun's plugin from one entry in its config, vitest.config.ts or vite.config.ts. Where the config has plugins already, add the entry inside that list:

import { defineConfig } from 'vitest/config'

export default defineConfig({
  plugins: [process.env.HUSHRUN_VITEST ? import(process.env.HUSHRUN_VITEST).then((m) => m.default()) : undefined],
})

Jest loads hushrun's file through NODE_OPTIONS, so no file changes. A package.json script that sets NODE_OPTIONS without what it had drops it. Keep it (NODE_OPTIONS="$NODE_OPTIONS …"), or put this line first in jest.config.js:

if (process.env.HUSHRUN_JEST) require(process.env.HUSHRUN_JEST);

For a jest.config.ts that compiles to CommonJS:

import { createRequire } from 'node:module'; if (process.env.HUSHRUN_JEST) createRequire(__filename)(process.env.HUSHRUN_JEST);

For a config that is an ES module, the same line with createRequire(import.meta.url) in place of createRequire(__filename).

Gradle loads hushrun's script from one line in settings.gradle, after any pluginManagement {} and plugins {} block:

if (System.getenv("HUSHRUN_GRADLE")) apply from: System.getenv("HUSHRUN_GRADLE")

Or in settings.gradle.kts:

System.getenv("HUSHRUN_GRADLE")?.let { apply(from = it) }

Rust's cargo test and cargo nextest run reuse whole test binaries, never one test: with unit: test, the run says so once and goes on by test binary.

Watch first with observe

Start with observe. Every test runs, and hushrun records what each one read and says what reuse would have left out. Nothing is left out yet.

checks:
  unit:
    tests:
      mode: observe
      unit: test

From the second run on, before the runner starts, a line says what reuse would have done:

hushrun: unit: with `tests: reuse`, 2 of 13 tests would have run, in 2 of 5 test files; 11 already passed on the files they read

Make a few ordinary changes and read the line each time. If it says every test would have run, it says why, and the reasons are in Limits.

Turn on reuse

Change observe to reuse:

checks:
  unit:
    tests:
      mode: reuse
      unit: test

Now the run leaves out the tests that already passed on the files they read. A few of them run anyway, as a check, and the line counts them:

hushrun: unit: 3 of 101 tests run, in 2 of 2 test files; 98 already passed on the files they read, and 2 more that already passed run anyway, as a check

The runner prints its own output for the tests it ran, and the run's last line says what reusing saved:

hushrun: unit passed in 3s; reusing 98 tests saved about 1s; it won't run again on this code

To reuse whole test files instead, leave out unit: test, or write tests: reuse. The line then counts test files. Each runner has its own word for them: test classes for Gradle and Maven, packages for Go, test targets for swift test, test programs for CTest.

Ask why a test ran

hushrun why says what a run would do with one test, for the files on disk now, and why. Name the test as its test file, ::, and its name:

hushrun why unit tests/test_cart.py::test_one

tests/test_cart.py::test_one would be reused: it passed on the files it read, and none of them has changed since

tests/test_cart.py::test_flips runs: its newest result on the files it reads now failed

It writes nothing. Without the :: part, it answers for a whole test file. All its options: Commands.

What keeps it safe

A test is left out only when it passed on the same files it reads now. Three more rules keep a wrong pass from spreading:

• A failed run keeps nothing it can't explain. When a test file fails and none of its tests did, as with a failing afterAll or a crash between tests, none of that file's tests keeps a pass from the run. Something hushrun couldn't follow inside a test, such as a process the test started, keeps that test from being reused.
• An audit reruns a small share. Of the tests a run could leave out, 2% run anyway, at least one. One that fails, though it passed before on the same files, runs once more. If it fails again, reuse is suspended for the check: every test runs until a person turns it back on with hushrun tests resume unit. A coding agent can't.
• A build that is out of date records nothing. When a test ran code older than its sources, as a CTest program built before an edit, the run keeps its exit code and records no pass:

hushrun: unit passed, but it isn't recorded: t_misc ran old code: tests/t_misc.c changed after CMakeFiles/t_misc.dir/tests/t_misc.c.o was made; build before ctest

The audit's lines:

hushrun: unit: tests/test_two.py::test_ok failed, though it passed before on the same files it read: running it once more
hushrun: unit: tests/test_two.py::test_ok failed again, though it passed before on the same files it read: reuse of unit's test files is suspended

Where it pays

It pays when a change reaches few tests:

• One test file changed. On open-source libraries it was tried on, a change to one test file ran 144 of click's 2,651 tests, 68 of httpx's 1,412 and 35 of luxon's 1,222.
• One file that few tests read changed. A change to one file ran 13 of gin's 641 tests, and 13 of swift-argument-parser's 569.

Collecting, importing and building still happen, even for one test of a file. So a run that leaves out 95% of the tests saves less than 95% of the time.

The trades

What you give up, and when it saves nothing:

• A change outside the sources runs every test. A README, a changelog, a CI file or a lockfile may be read by any test, so changing one runs them all. On the libraries it was tried on, nearly every commit touched one. List the files no test reads in tests.ignore.
• One shared entry point means every test reads everything. Where tests import the package through one __init__.py, or an index file that re-exports all of it, every test reads every module, and a change to any source runs them all.
• reads: test costs much more and saved little. With reads: test, hushrun watches what each test itself calls and opens. On the libraries it was tried on, watching added 10 to 46% to a pytest, Vitest or Jest run with reads: file, and 81 to 453% with reads: test, which then ran only 2 to 10% fewer tests after a change. It works for pytest on Python 3.12 or later, Vitest and Jest; the other runners go on with reads: file, and say so. Try it in observe before you keep it.
• Tests of one file now run apart. Tests that pass only together, through shared state or order, can fail when one runs alone. Use unit: file for that check, or list the file in tests.always.
• Reports hold only what ran. A run that left tests out writes test reports for the tests it ran, and its coverage isn't recorded.

The line for a change outside the sources:

hushrun: unit: all 8 test files run: requirements.txt changed, and any test may read it

Limits

Some set-ups can't be watched. hushrun then runs every test and says why once:

• Vitest with typecheck on, isolate: false, or tests in a browser; vitest without run, which watches. With Vitest's coverage on, no test is left out.
• Jest with its coverage on runs every test file. Where ts-jest checks types with its language service, every test file reads every file of code.
• Go: a COMMAND with -run or -skip reuses by package, and so does a package with a TestMain. A package that failed keeps its other tests' results only when go test names each one: add -v.
• Gradle: Kotest's tests aren't left out one by one. A JUnit 4 task leaves out at most 250 tests. Switching between a plain Gradle run and one through hushrun reruns the test tasks, each way. With Isolated Projects on, nothing is watched.
• Maven: JUnit 5 needs Surefire 3.0.0-M4 or later, and JUnit 4 and TestNG Surefire 3.6 or later. mvnd isn't watched.
• swift test: not with --enable-code-coverage, --skip-build, xcrun swift test or a path to swift, and not on Windows.
• CTest: make test, ninja test and cmake --build … --target test start CTest by its full path, and aren't watched. Build before you run ctest.
• pytest: under tox or nox, or a Python started with -I or -E, hushrun loads nothing, and the run says why.
• Tests with tight timing can fail more often under hushrun run than plain, in every mode, since hushrun adds processes and pipes.

Every setting: tests in Configuration.

Next

• How it works: what hushrun compares before a whole check skips.
• Configuration: tests key by key.
• Day to day: run checks, and read what hushrun prints.