# Deploys on GitLab CI

This guide sets up a GitLab project that deploys from `main` through hushrun's pipeline. A push of files that already passed deploys the image built for them, by digest, with no build and no check to repeat. For what a deploy pipeline is, start at [Deploys](https://hushrun.com/docs/deploy.md).

## Before you begin

- The Maintainer role on the project, with `main` protected (Settings, Repository, Protected branches). Protected variables reach only protected branches, so the token and key reach only your deploy pipelines.
- GitLab 17.0 or later, on GitLab.com or self-managed. It runs a job's `after_script` when it cancels the job, which is how the pipeline records a cancel.
- An org on hushrun ([sign up](https://app.hushrun.com/sign-up)), with the project added: an org admin's `npx hushrun init` adds it.
- An org admin, to make the pipeline's agent token.
- The container registry your image already uses.
- Linux runners.

A GitLab pipeline signs as one of the org's agents, not as the project's CI. See [How a GitLab pipeline signs](#signing).

## Write hushrun.yaml

```bash
hushrun init --pipeline --image us-docker.pkg.dev/acme/api/server
```

It recognizes a GitLab project by its `.gitlab-ci.yml`, or by its host, such as `gitlab.com` or `gitlab.example.com`. Edit the file: the checks the deploy requires and their commands, the branch, and in a monorepo the scopes and `digest.exclude` ([What a deploy requires](https://hushrun.com/docs/deploy.md#deploy)).

Each scope covers `.gitlab-ci.yml` and anything it includes, so that a change to the pipeline changes the digest.

## Connect the project

Run `npx hushrun init`. It adds `repository_url` and `transparency_log` to `hushrun.yaml`. Commit the file: hushrun reads it as committed, and the pipeline takes the repository's URL from it.

## Make the pipeline's agent token

As an org admin, in the project, at a terminal:

```bash
hushrun token create --agent gitlab-ci --expires 365d
```

It prints the token, `hra_…`, and the agent's machine key, a PEM block, once, and keeps the key out of anything that could be a log. In a script, `--format json` gives them. The `HUSHRUN_AGENT` string it also shows is for sandboxes, not pipelines.

A token lasts 30 days unless `--expires` says otherwise, 365 days at most. Make the next one before it expires ([Day to day](#day-to-day)).

## Set the CI/CD variables

In Settings, CI/CD, Variables:

| Variable | Type | Flags | Value |
|---|---|---|---|
| `HUSHRUN_TOKEN` | Variable | Masked, protected | The agent's token, `hra_…` |
| `HUSHRUN_MACHINE_KEY_FILE` | File | Protected | The agent's machine key: the whole PEM block |
| `HUSHRUN_GITLAB_TOKEN` | Variable | Masked | Optional: a project access token with `read_api`, for `hushrun doctor` |

On the same page, set **Minimum role to use pipeline variables** to **No one allowed**, or to **Maintainer** if Maintainers force full runs. Variables set when someone runs a pipeline outrank project variables, so whoever may set them could point the pipeline at another server, or change how it signs.

- The key is a File variable because GitLab can't mask a PEM key. GitLab writes it to a file outside the repository, and `HUSHRUN_MACHINE_KEY_FILE` holds the file's path. Every job signs with it.
- With `HUSHRUN_GITLAB_TOKEN`, `hushrun doctor` checks that `main` is protected, and who may set pipeline variables. Nothing else uses it: plans never ask GitLab about pipelines. Make it in Settings, Access tokens. Reading who may set pipeline variables takes the Maintainer role.
- The pipeline needs no `HUSHRUN_REPOSITORY_URL`. Set one only to pin the URL in the project's settings too: it must then match `hushrun.yaml`'s, or commands exit 2.
- On a hushrun server of your own, set `HUSHRUN_HOST` to its origin, such as `https://hushrun.example.com`, so that the agent's token goes there.

## Add the pipeline

1. Take `install-hushrun.sh` from a hushrun release, read it, and commit it as `ci/install-hushrun.sh`. It installs a release for the runner's OS and architecture, after checking the archive's SHA-256 ([details](#hushrun-version)).
2. Add this `.gitlab-ci.yml` at the project's root, and set:
   - the `test` job's image: the toolchain your checks need;
   - the `deploy` job's image and scripts: `./build.sh "$HUSHRUN_REF"` pushes the image to `$HUSHRUN_REF`, and `./deploy.sh "$HUSHRUN_REF"` deploys it, pinned by digest;
   - your cloud's credential step for the registry and the deploy, in `.hushrun` ([Cloud access](#cloud-access)).

```yaml title=".gitlab-ci.yml"
# Names, images and the build and deploy scripts are placeholders. The CI/CD variables are
# the project's, never set here (hushrun.com/docs/deploy/gitlab#variables).

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
    - when: never

stages: [plan, test, deploy, finish]

# The newest hushrun release: the plan job runs the image's, and every later job the same one,
# even when a newer one comes out meanwhile. To install one release until you change it, set
# HUSHRUN_VERSION, and the image to that release, pinned by the digest its release notes give:
#   HUSHRUN_VERSION: "0.9.0"
#   HUSHRUN_IMAGE: ghcr.io/hushrun/hushrun:0.9.0@sha256:…
variables:
  HUSHRUN_IMAGE: ghcr.io/hushrun/hushrun:latest

# Every job that runs hushrun extends this. GitLab's `extends` replaces a before_script rather
# than merging it, so your cloud's credential step for the registry, if it needs one, goes
# here too (https://hushrun.com/docs/deploy/gitlab#cloud-access).
.hushrun:
  before_script:
    # The image has hushrun. On any other image, or when it isn't HUSHRUN_VERSION, the release
    # the plan job ran (from its dotenv report), install that release, checked against its
    # checksum.
    - |
      have=$(hushrun --version 2>/dev/null || true)
      if [ -z "$have" ] || { [ -n "${HUSHRUN_VERSION:-}" ] && [ "$have" != "hushrun $HUSHRUN_VERSION" ]; }; then
        bin=$(sh ci/install-hushrun.sh)
        PATH="$(dirname "$bin"):$PATH"
      fi
  # A cancel records `superseded`: GitLab runs after_script, in a shell of its own, when it
  # cancels a running job. A timeout is a failure, which mark-failed records.
  after_script:
    - |
      if [ "$CI_JOB_STATUS" = canceled ] && [ -f plan.json ]; then
        case "$HUSHRUN_MODE" in
          promote | refresh | full)
            have=$(hushrun --version 2>/dev/null || true)
            if [ -z "$have" ] || { [ -n "${HUSHRUN_VERSION:-}" ] && [ "$have" != "hushrun $HUSHRUN_VERSION" ]; }; then
              bin=$(sh ci/install-hushrun.sh)
              PATH="$(dirname "$bin"):$PATH"
            fi
            hushrun mark superseded --plan plan.json --reason cancelled || true
            ;;
        esac
      fi

plan:
  stage: plan
  extends: .hushrun
  image: $HUSHRUN_IMAGE
  script:
    - hushrun plan --start --out plan.json --dotenv plan.env
    # Later jobs run this release, the one that made the plan.
    - echo "HUSHRUN_VERSION=$(hushrun --version | cut -d ' ' -f 2)" >> plan.env
  artifacts:
    paths: [plan.json]
    reports:
      dotenv: plan.env # HUSHRUN_MODE, HUSHRUN_CHECKS, HUSHRUN_DIGEST, HUSHRUN_REF, …, HUSHRUN_VERSION

test:
  stage: test
  extends: .hushrun
  image: golang:1.25 # the toolchain the checks need
  needs: [plan]
  script:
    - '[ -n "$HUSHRUN_CHECKS" ] || { echo "nothing to run"; exit 0; }'
    - hushrun run --plan plan.json # exactly the plan's checks

deploy:
  stage: deploy
  extends: .hushrun
  image: google/cloud-sdk:slim # what build.sh and deploy.sh need
  needs: [plan, test]
  resource_group: deploy # one deploy at a time
  script:
    - 'case "$HUSHRUN_MODE" in promote|refresh|full) ;; *) echo "mode $HUSHRUN_MODE: nothing to deploy"; exit 0 ;; esac'
    # Each output goes through a variable, so a command that fails fails the job.
    - out=$(hushrun guard --plan plan.json --format env); eval "$out"
    - '[ "$HUSHRUN_STALE" != true ] || { echo "a newer revision took over"; exit 0; }'
    - out=$(hushrun artifact resolve --plan plan.json --format env); eval "$out"
    - |
      if [ "$HUSHRUN_BUILD" = true ]; then
        ./build.sh "$HUSHRUN_REF"
        out=$(hushrun attest --plan plan.json --artifact "image=$HUSHRUN_REF" --format env)
        eval "$out"
      fi
    - ./deploy.sh "$HUSHRUN_REF"
    - hushrun mark deployed --plan plan.json --artifact "image=$HUSHRUN_REF"
    # A check of what was just deployed, such as a smoke test, runs here: its failure fails the
    # job, and mark-failed records it.
    - hushrun mark passed --plan plan.json --artifact "image=$HUSHRUN_REF"

mark-failed:
  stage: finish
  extends: .hushrun
  image: $HUSHRUN_IMAGE
  when: on_failure
  script:
    - 'case "$HUSHRUN_MODE" in promote|refresh|full) ;; *) exit 0 ;; esac' # nothing started
    - hushrun mark failed --plan plan.json --reason "pipeline failed"
```

[The pipeline's jobs](#jobs) explains each job.

## Check the setup

From a job, or on your machine with `HUSHRUN_GITLAB_TOKEN` and `CI_PROJECT_PATH` set (and `CI_API_V4_URL` on a self-managed GitLab):

```bash
hushrun doctor
```

It exits 2 when a check fails. Besides what it checks everywhere ([Commands](https://hushrun.com/docs/reference/cli.md#hushrun-doctor)), on GitLab it checks that `HUSHRUN_GITLAB_TOKEN` can read the project, that `main` is protected, and who may set pipeline variables. In a job, it checks that the server lists the job's machine key for the agent.

## Push to main

Push to `main`. The first pipeline plans `full`: it runs the checks, builds, deploys, and records each step. Run a new pipeline for the same files, or later revert to them, and it plans `promote`: it deploys the image it already built and checked, by digest, with no build and no check to repeat. The plan job's log starts with the mode and the reason.

## Day to day

- **`blocked`.** The pipeline fails on purpose, and the plan job's log says what to do: the pipeline runs on another branch than `deploy.branch`, or it was forced where `allow_force: false` forbids it.
- **`current`.** The project already runs this code. To deploy it again, run a new pipeline for `main` by hand. Retrying the plan job doesn't: a `current` plan records nothing, so the retry is still a first attempt.
- **`refresh`.** Only the checks without valid evidence run ([Which checks run](https://hushrun.com/docs/deploy.md#checks-to-run)), and the image is still reused.
- **Force a full run.** A Maintainer runs the pipeline for `main` with `HUSHRUN_FORCE_FULL=true`, which needs **Minimum role to use pipeline variables** set to **Maintainer**. A forced run builds and runs every required check.
- **Retry the plan job.** GitLab keeps the pipeline's id, so `hushrun plan` counts attempts itself: one more than the pipeline recorded. The other jobs take the plan's attempt from `plan.json`.
- **A newer commit landed while deploying.** `guard` records `superseded`, and the deploy job stops. The newer pipeline deploys instead.
- **Renew the token.** Make the next one under the same agent name, which comes with a new key, and replace `HUSHRUN_TOKEN` and `HUSHRUN_MACHINE_KEY_FILE`. Then revoke the old one with `hushrun token revoke hra_XXXXXX`, its prefix from `hushrun token list`. What the old key signed keeps counting.
- **The org's plan is used up.** New check results aren't recorded, and `run` and `attest` say so once, with where to upgrade. Everything else is recorded, and deploys go on: see [Plans and limits](https://hushrun.com/docs/plans.md#at-the-limit).

Exit codes are the same as on GitHub: see [Exit codes](https://hushrun.com/docs/deploy/github.md#exit-codes).

## How a GitLab pipeline signs

hushrun's server verifies GitHub Actions' ID tokens, not GitLab's. So a GitLab pipeline signs and writes as one of the org's agents, with two CI/CD variables:

- **The agent's token**, `HUSHRUN_TOKEN`. hushrun sends it to the repository's URL that `hushrun.yaml` names, because that URL is on hushrun's host, which takes an agent token for its org only. Other tokens, such as `HUSHRUN_REPOSITORY_TOKEN`, never go to a URL that only a file in the repository names.
- **The machine key** that came with the token, `HUSHRUN_MACHINE_KEY_FILE`. It signs every event, which counts as `agent:<name>` until an admin revokes the token. What it signed before keeps counting.

[CI reuses only CI's results](https://hushrun.com/docs/deploy.md#ci-only) counts only the repository's own CI, so don't turn it on for a GitLab project: its pipeline's events wouldn't count.

## The pipeline's jobs

- **plan** decides once. It records `started`, and passes its outputs to later jobs as a dotenv report: `HUSHRUN_MODE`, `HUSHRUN_CHECKS`, `HUSHRUN_DIGEST`, `HUSHRUN_REF` and more, and the release it ran, `HUSHRUN_VERSION`. The plan itself, which the report leaves out, travels as the artifact `plan.json`. A `blocked` plan exits 3, which fails the pipeline on purpose.
- **test** runs exactly the plan's checks, and exits 8 when one fails.
- **deploy** runs only when the mode is `promote`, `refresh` or `full`:
  1. `guard` stops a deploy of a revision that is no longer the branch's head.
  2. `artifact resolve` gives the image to deploy, pinned by digest, or asks for a build.
  3. When the plan builds, your build runs, and `attest` records the image.
  4. Your deploy runs, then `mark deployed`. A check of what was just deployed, such as a smoke test, goes here.
  5. `mark passed`.
- **mark-failed** records `failed` when a job failed or timed out. A job GitLab cancels records `superseded` from the `.hushrun` template's `after_script`.

Two rules hold in every job:

- A command whose output a job evaluates goes through a variable, `out=$(hushrun guard … --format env); eval "$out"`, because `eval "$(hushrun guard …)"` succeeds when hushrun fails.
- Every job carries out the plan for the pipeline's commit: never check out another ref, and never pass `--rev`.

### Which hushrun the pipeline runs

Jobs that only run hushrun use the image `ghcr.io/hushrun/hushrun`: Alpine, git, CA certificates and hushrun's static binary, with no entrypoint. Its `latest` tag is the newest release, and each release has its version's tag.

The plan job runs `latest` and passes its version to the later jobs, which install that same release with `install-hushrun.sh`. A new release reaches your pipelines at their next run: its fixes without an edit, and a bad release as fast.

- To pin a release, set `HUSHRUN_VERSION`, such as `"0.9.0"`, and `HUSHRUN_IMAGE` to that release, pinned by the digest its release notes give. A pinned version changes only through a commit you review.
- The script checks the archive against the checksum it pins for that release. For a release it doesn't list, such as a newer one, it uses the checksum published with the release.
- Runners that can't reach GitHub need a pinned version. Mirror the image and the release archives where they can read them, and point `HUSHRUN_IMAGE` and `HUSHRUN_DOWNLOAD_URL` there.

## Cloud access

hushrun's server needs no cloud credentials. Your registry and your deploy do:

- Give the jobs those roles through your cloud's workload identity federation with GitLab's ID tokens, with no long-lived cloud key.
- Bind the access to the project's id and a protected ref.
- Put the credential step in `.hushrun`'s `before_script`: a job's own `before_script` would replace the template's.

## Merge requests

In a merge request's pipeline, hushrun reads `hushrun.yaml` from the branch the merge request targets, which was reviewed, never from the change. Fetch that branch and name it with `--config-ref`, or every command that reads or writes events exits 2:

```bash
git fetch --depth=1 origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
hushrun run --all --config-ref "origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
```

- The token and key are protected variables, so they don't reach a merge request pipeline from an unprotected branch. The pipeline on this page runs hushrun's jobs on `main` only.
- A merge request pipeline has no branch, so it never deploys. The checks it passes count for the same digest: when `main` hasn't moved, the merged commit has the same files, and the deploy doesn't run them again.

## Limits

- **Child pipelines** record runs under their own pipeline id: run hushrun's jobs in the parent pipeline.
- **Windows runners**: the install script is for Linux and macOS.
- **`hushrun adopt`** writes GitHub workflows only: start a GitLab project from the pipeline on this page.
- **Keyless signing** with GitLab's ID tokens (`id_tokens: {SIGSTORE_ID_TOKEN: {aud: sigstore}}` and `HUSHRUN_SIGNING: sigstore`) counts as the project's CI only on a server that verifies GitLab's tokens for the project. hushrun's server doesn't, so sign with the agent's key there.

## Next

- [The planner](https://hushrun.com/docs/deploy.md#planner): what each mode means, and what happens in common situations.
- [Evidence from agents and laptops](https://hushrun.com/docs/deploy/evidence.md): let agents run the deploy's checks before a push.
- [Transparency log](https://hushrun.com/docs/deploy/transparency.md): monitor hushrun's server from a scheduled pipeline.