# Reuse tests one by one

When a check's code changed, `hushrun run` normally runs every test again. With `tests` set in `hushrun.yaml`, it leaves out the tests that already passed on the files they read, and runs the rest. It's off until you turn it on, check by check.

```text
hushrun: unit: 3 of 101 tests run, in 2 of 2 test files; 98 already passed on the files they read, and 2 more that already passed run anyway, as a check
```

It leaves out whole test files, or each test on its own. It starts after a miss: when the check's [result key](https://hushrun.com/docs/run.md#same-code) has a pass, the check skips as before, and no test runs.

## Before you begin

- hushrun 0.16.0 or later, for tests one by one. Before it, hushrun reuses only whole test files.
- A check run as `hushrun run NAME -- COMMAND`, where COMMAND starts one of the test runners below. The check's settings go under its `NAME` in `hushrun.yaml`, which `npx hushrun init` writes.
- Results on: an organization's backend, or this machine's cache ([Where results go](https://hushrun.com/docs/results.md)). With results off, hushrun looks at no test.

## Check what your runner needs

Name the runner in COMMAND, as below. A script that hides it, such as `npm test` or `make test`, gets no test left out, except for pytest, found wherever it runs, and CTest and `swift test`, found in a shell line, a script or a Makefile that starts them by name.

| Runner | COMMAND | What to add |
|---|---|---|
| pytest | `pytest`, `python -m pytest`, or anything that starts pytest | Nothing |
| Vitest | `vitest run`, `npx vitest run`, `pnpm vitest run` | One line in Vitest's config, below |
| Jest 29 or 30 | `jest`, `npx jest`, `pnpm jest` | Nothing, unless a script replaces `NODE_OPTIONS`: below |
| Go | `go test ./...` | Nothing. Go 1.20 or later leaves out one test; older Go leaves out whole packages |
| Gradle: Java, Kotlin, Android's local unit tests | `./gradlew test`, `gradle test` | One line in the settings file, below |
| Maven | `mvn test`, `./mvnw verify` | Nothing |
| `swift test` | `swift test` | Nothing. Swift 6.0 or later, on macOS or Linux |
| CTest, for C and C++ | `ctest --test-dir build` | Nothing. CTest 3.21 or later |

`hushrun init` proposes the lines for Vitest, Jest and Gradle in its list of changes to make by hand. It never writes them. Without `HUSHRUN_VITEST`, `HUSHRUN_JEST` or `HUSHRUN_GRADLE`, which hushrun sets for its own runs, each line does nothing.

**Vitest** loads hushrun's plugin from one entry in its config, `vitest.config.ts` or `vite.config.ts`. Where the config has `plugins` already, add the entry inside that list:

```ts title="vitest.config.ts"
import { defineConfig } from 'vitest/config'

export default defineConfig({
  plugins: [process.env.HUSHRUN_VITEST ? import(process.env.HUSHRUN_VITEST).then((m) => m.default()) : undefined],
})
```

**Jest** loads hushrun's file through `NODE_OPTIONS`, so no file changes. A `package.json` script that sets `NODE_OPTIONS` without what it had drops it. Keep it (`NODE_OPTIONS="$NODE_OPTIONS …"`), or put this line first in `jest.config.js`:

```js title="jest.config.js"
if (process.env.HUSHRUN_JEST) require(process.env.HUSHRUN_JEST);
```

For a `jest.config.ts` that compiles to CommonJS:

```ts title="jest.config.ts"
import { createRequire } from 'node:module'; if (process.env.HUSHRUN_JEST) createRequire(__filename)(process.env.HUSHRUN_JEST);
```

For a config that is an ES module, the same line with `createRequire(import.meta.url)` in place of `createRequire(__filename)`.

**Gradle** loads hushrun's script from one line in `settings.gradle`, after any `pluginManagement {}` and `plugins {}` block:

```
if (System.getenv("HUSHRUN_GRADLE")) apply from: System.getenv("HUSHRUN_GRADLE")
```

Or in `settings.gradle.kts`:

```
System.getenv("HUSHRUN_GRADLE")?.let { apply(from = it) }
```

Rust's `cargo test` and `cargo nextest run` reuse whole test binaries, never one test: with `unit: test`, the run says so once and goes on by test binary.

## Watch first with observe

Start with `observe`. Every test runs, and hushrun records what each one read and says what reuse would have left out. Nothing is left out yet.

```yaml title="hushrun.yaml"
checks:
  unit:
    tests:
      mode: observe
      unit: test
```

From the second run on, before the runner starts, a line says what `reuse` would have done:

```text
hushrun: unit: with `tests: reuse`, 2 of 13 tests would have run, in 2 of 5 test files; 11 already passed on the files they read
```

Make a few ordinary changes and read the line each time. If it says every test would have run, it says why, and the reasons are in [Limits](#limits).

## Turn on reuse

Change `observe` to `reuse`:

```yaml title="hushrun.yaml"
checks:
  unit:
    tests:
      mode: reuse
      unit: test
```

Now the run leaves out the tests that already passed on the files they read. A few of them run anyway, as a check, and the line counts them:

```text
hushrun: unit: 3 of 101 tests run, in 2 of 2 test files; 98 already passed on the files they read, and 2 more that already passed run anyway, as a check
```

The runner prints its own output for the tests it ran, and the run's last line says what reusing saved:

```text
hushrun: unit passed in 3s; reusing 98 tests saved about 1s; it won't run again on this code
```

To reuse whole test files instead, leave out `unit: test`, or write `tests: reuse`. The line then counts test files. Each runner has its own word for them: test classes for Gradle and Maven, packages for Go, test targets for `swift test`, test programs for CTest.

## Ask why a test ran

`hushrun why` says what a run would do with one test, for the files on disk now, and why. Name the test as its test file, `::`, and its name:

```bash
hushrun why unit tests/test_cart.py::test_one
```

```text
tests/test_cart.py::test_one would be reused: it passed on the files it read, and none of them has changed since
```

```text
tests/test_cart.py::test_flips runs: its newest result on the files it reads now failed
```

It writes nothing. Without the `::` part, it answers for a whole test file. All its options: [Commands](https://hushrun.com/docs/reference/cli.md).

## What keeps it safe

A test is left out only when it passed on the same files it reads now. Three more rules keep a wrong pass from spreading:

- **A failed run keeps nothing it can't explain.** When a test file fails and none of its tests did, as with a failing `afterAll` or a crash between tests, none of that file's tests keeps a pass from the run. Something hushrun couldn't follow inside a test, such as a process the test started, keeps that test from being reused.
- **An audit reruns a small share.** Of the tests a run could leave out, 2% run anyway, at least one. One that fails, though it passed before on the same files, runs once more. If it fails again, reuse is suspended for the check: every test runs until a person turns it back on with `hushrun tests resume unit`. A coding agent can't.
- **A build that is out of date records nothing.** When a test ran code older than its sources, as a CTest program built before an edit, the run keeps its exit code and records no pass:

```text
hushrun: unit passed, but it isn't recorded: t_misc ran old code: tests/t_misc.c changed after CMakeFiles/t_misc.dir/tests/t_misc.c.o was made; build before ctest
```

The audit's lines:

```text
hushrun: unit: tests/test_two.py::test_ok failed, though it passed before on the same files it read: running it once more
hushrun: unit: tests/test_two.py::test_ok failed again, though it passed before on the same files it read: reuse of unit's test files is suspended
```

## Where it pays

It pays when a change reaches few tests:

- **One test file changed.** On open-source libraries it was tried on, a change to one test file ran 144 of click's 2,651 tests, 68 of httpx's 1,412 and 35 of luxon's 1,222.
- **One file that few tests read changed.** A change to one file ran 13 of gin's 641 tests, and 13 of swift-argument-parser's 569.

Collecting, importing and building still happen, even for one test of a file. So a run that leaves out 95% of the tests saves less than 95% of the time.

## The trades

What you give up, and when it saves nothing:

- **A change outside the sources runs every test.** A README, a changelog, a CI file or a lockfile may be read by any test, so changing one runs them all. On the libraries it was tried on, nearly every commit touched one. List the files no test reads in `tests.ignore`.
- **One shared entry point means every test reads everything.** Where tests import the package through one `__init__.py`, or an index file that re-exports all of it, every test reads every module, and a change to any source runs them all.
- **`reads: test` costs much more and saved little.** With `reads: test`, hushrun watches what each test itself calls and opens. On the libraries it was tried on, watching added 10 to 46% to a pytest, Vitest or Jest run with `reads: file`, and 81 to 453% with `reads: test`, which then ran only 2 to 10% fewer tests after a change. It works for pytest on Python 3.12 or later, Vitest and Jest; the other runners go on with `reads: file`, and say so. Try it in `observe` before you keep it.
- **Tests of one file now run apart.** Tests that pass only together, through shared state or order, can fail when one runs alone. Use `unit: file` for that check, or list the file in `tests.always`.
- **Reports hold only what ran.** A run that left tests out writes test reports for the tests it ran, and its coverage isn't recorded.

The line for a change outside the sources:

```text
hushrun: unit: all 8 test files run: requirements.txt changed, and any test may read it
```

## Limits

Some set-ups can't be watched. hushrun then runs every test and says why once:

- **Vitest** with `typecheck` on, `isolate: false`, or tests in a browser; `vitest` without `run`, which watches. With Vitest's coverage on, no test is left out.
- **Jest** with its coverage on runs every test file. Where ts-jest checks types with its language service, every test file reads every file of code.
- **Go:** a COMMAND with `-run` or `-skip` reuses by package, and so does a package with a `TestMain`. A package that failed keeps its other tests' results only when `go test` names each one: add `-v`.
- **Gradle:** Kotest's tests aren't left out one by one. A JUnit 4 task leaves out at most 250 tests. Switching between a plain Gradle run and one through hushrun reruns the test tasks, each way. With Isolated Projects on, nothing is watched.
- **Maven:** JUnit 5 needs Surefire 3.0.0-M4 or later, and JUnit 4 and TestNG Surefire 3.6 or later. `mvnd` isn't watched.
- **`swift test`:** not with `--enable-code-coverage`, `--skip-build`, `xcrun swift test` or a path to `swift`, and not on Windows.
- **CTest:** `make test`, `ninja test` and `cmake --build … --target test` start CTest by its full path, and aren't watched. Build before you run `ctest`.
- **pytest:** under tox or nox, or a Python started with `-I` or `-E`, hushrun loads nothing, and the run says why.
- Tests with tight timing can fail more often under `hushrun run` than plain, in every mode, since hushrun adds processes and pipes.

Every setting: [`tests` in Configuration](https://hushrun.com/docs/reference/configuration.md#tests).

## Next

- [How it works](https://hushrun.com/docs/run.md): what hushrun compares before a whole check skips.
- [Configuration](https://hushrun.com/docs/reference/configuration.md#tests): `tests` key by key.
- [Day to day](https://hushrun.com/docs/daily.md): run checks, and read what hushrun prints.