# CI/CD for AI agents

"CI/CD for AI agents" can mean two jobs: shipping an agent you built, or keeping CI working while coding agents write your code. They share a name and little else, so start with which one you have.

| You're | The hard part | Read |
|---|---|---|
| Building an agent that runs in production, such as a support agent | Its behavior can change with no code change, and its tests cost money | This page |
| Using coding agents on your repository | They push more often than CI can keep up with | [Keep CI useful with coding agents](https://hushrun.com/docs/guides/coding-agents-ci.md) |

The rest of this page covers the first job.

## What's different about an agent

A unit test gives the same answer for the same code. An agent may not, and that changes four things:

- **More than code is the release.** Prompts, the model, tool definitions and retrieved data all change what the agent does.
- **Tests are probabilistic.** The same input can give different answers, so one failed case doesn't prove a regression.
- **Tests cost money.** Each eval case calls a model, so you pay for tokens as well as CI minutes.
- **Behavior can change with no commit.** Retrieved data changes, a model behind an alias changes, and a provider's serving can change under a pinned model.

## Keep the release in the repository

What isn't in the repository can't be reviewed in a pull request, doesn't roll back with the code, and doesn't start your checks when it changes.

- **Prompts:** files, reviewed like code.
- **The model:** an exact ID in a file, not an alias that moves. Check what your provider promises. [Anthropic says](https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions) a Claude model ID's weights stay fixed, and that infrastructure updates can still make minor differences.
- **Tools:** their definitions and schemas.
- **Eval cases:** a fixed set, with each production failure added as a new case.

## Run checks in order of cost

| Check | Costs | When |
|---|---|---|
| Lint, types, tool schemas | CI minutes | Every push |
| Unit tests with the model mocked | CI minutes | Every push |
| Evals on a fixed set | Minutes and tokens | When the agent's code, prompts or model change |
| Full evals | Minutes and the most tokens | On a schedule |

Evals don't need to run on every commit: [Microsoft's Foundry docs](https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluation-github-action) advise against it, to keep costs down.

Keep the schedule even when nothing was committed. On those days it's the only eval that runs, so it's what catches a change outside the repository. In a public repository, [GitHub disables scheduled workflows](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows) after 60 days with no activity.

## Gate on a pass rate

One failed case can be noise, so judge the whole set against a threshold, and make the job a required check so that falling below it blocks the merge. Set the threshold from what the agent scores today, before you see a new change's results.

[promptfoo](https://www.promptfoo.dev/docs/usage/command-line/), for one, exits with code 100 when the pass rate falls below `PROMPTFOO_PASS_RATE_THRESHOLD`, so the step fails on its own:

```yaml title=".github/workflows/evals.yml"
on:
  pull_request:
  schedule:
    - cron: "0 3 * * *"

jobs:
  evals:
    # Forks don't get your secrets, so their pull requests can't run this.
    if: github.event_name == 'schedule' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npx promptfoo eval --no-cache
        env:
          PROMPTFOO_PASS_RATE_THRESHOLD: "95"
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

It needs a committed lockfile, `promptfoo` in `devDependencies`, a promptfoo config, and the key as a repository secret.

- **It runs on every push to a pull request.** To run it only when the agent's files change, [filter inside the job](https://hushrun.com/docs/guides/github-actions-run-when-files-change.md#job). A `paths` filter on the trigger would leave a required check [waiting for a status](https://hushrun.com/docs/guides/required-check-waiting-for-status.md).
- **The key goes to that step alone,** so the job's other steps don't get it. Everything that step runs still can, including dependencies and code from the pull request.
- **`--no-cache` makes the step call the model.** promptfoo caches responses on disk, and a cached answer can't show that the model behind an ID changed.
- **A skipped job reports success,** so a fork's pull request passes this check with no evals. Run them yourself before you merge one.

## Ship the artifact you tested

An agent is still software. Build one image and deploy that image by digest, so production runs the code and prompts your evals saw. See [Build once, deploy many](https://hushrun.com/docs/guides/build-once-deploy-many.md). The sample above evaluates a checkout: to evaluate the image itself, point the evals at it running.

- **A prompt in the image rolls back with the code.** Deploying the previous digest restores both.
- **The model's weights aren't in the image.** Keep the exact ID in the image so that it rolls back too. Retrieved data doesn't roll back.
- **Roll out gradually.** Evals cover the cases you thought of, so watch cost, tool errors and how often people give up. Going back to the previous digest helps when the release caused the problem, not when the model or the data did.

## With hushrun

Evals are often the most expensive check to repeat, so skipping them can save the most. [`hushrun run`](https://hushrun.com/docs/run.md) skips a check when the same files, check name, command and directory already passed:

```json title="package.json"
"scripts": {
  "evals": "hushrun run -- promptfoo eval --no-cache"
}
```

In another language, it's the same idea: `hushrun run evals -- pytest evals/`.

The command is part of the check, so `promptfoo eval` and `promptfoo eval --no-cache` are two checks: use one everywhere. `promptfoo` and `hushrun` go in `devDependencies`.

Once `npx hushrun init` has set up a backend and [CI's ID token](https://hushrun.com/docs/ci.md), re-running a job, or the same code reaching CI again, spends no tokens on a pass you already have.

With no [scopes](https://hushrun.com/docs/run.md#monorepos), any file change runs the evals again, a prompt or a model ID included. If you narrow that with scopes or `digest.exclude`, keep your prompts and model settings inside them, or a changed prompt will skip.

Three limits matter more here than for ordinary tests:

- **A skip is about the files.** It says these files passed, not that the agent behaves the same today. A change behind a model ID or alias, data in a vector store, a prompt in a hosted registry and an environment variable are all outside what hushrun compares. So give the scheduled run `--force`, with the same name and command: `npx hushrun run --force evals -- promptfoo eval --no-cache`. It runs every time and records the result. An eval you run without hushrun runs too, but hushrun doesn't see it, so after it fails, the next wrapped run still skips.
- **A laptop's pass counts.** By default, a pass from a laptop or an agent's sandbox skips the check in CI, and `PROMPTFOO_PASS_RATE_THRESHOLD` isn't part of the comparison, since it's an environment variable. To make CI's gate the one that counts, turn on [CI reuses only CI's results](https://hushrun.com/docs/ci.md#ci-only).
- **A retry can pass.** A failed check always runs again, so retrying a probabilistic eval until it passes records that pass, and later runs skip. The dashboard marks a check that passed and failed on the same files as flaky. Treat that mark as a reason to look, and gate on a pass rate so that one run means something.

To gate a deploy on evals, use [promotion](https://hushrun.com/docs/promotion.md). It doesn't read `hushrun run`'s results, so declare the evals as a check in `hushrun.yaml`, run them with `hushrun run --stage`, and set `max_age`, or their evidence never expires. Production then gets the image staging built, copied by digest, with signed evidence of what passed. If a copy fails it falls back to a build, unless you pass `--no-fallback`. A stage can also require a person's approval, and hushrun refuses one from CI or an agent token.

## Related

- [Keep CI useful with coding agents](https://hushrun.com/docs/guides/coding-agents-ci.md)
- [Build once, deploy many](https://hushrun.com/docs/guides/build-once-deploy-many.md)
- [Promote a Docker image between environments](https://hushrun.com/docs/guides/promote-docker-image.md)