# Promotion on GitLab CI

This page sets up a GitLab project that deploys staging from `develop` and production from `main`, so that production deploys the image staging built and validated, copied by digest. For what promotion is, start at [Promotion](https://hushrun.com/docs/promotion.md).

## How a GitLab pipeline signs

hushrun's server verifies GitHub Actions' ID tokens, not GitLab's. So a GitLab pipeline signs and writes as one of the org's agents, with two of the project's CI/CD variables:

- **The agent's token**, `HUSHRUN_TOKEN`. hushrun sends it to the repository's URL that `hushrun.yaml` names, because that URL is on hushrun's host, which takes an agent token for its org only. Other tokens, such as `HUSHRUN_REPOSITORY_TOKEN`, never go to a URL that only a file in the repository names.
- **The machine key** registered with the token, `HUSHRUN_MACHINE_KEY_FILE`. It signs every event, and the events count as `agent:<name>` at every stage that isn't CI-only, until an admin revokes the token. Events it signed before keep counting.

A CI-only stage counts only the repository's own CI, so don't make a GitLab project's stages CI-only: its pipeline's events wouldn't count there.

## What you need

- The Maintainer role on the project, with `develop` and `main` protected (Settings, Repository, Protected branches). Protected variables reach only protected branches, so the agent's token and key reach only your stage pipelines.
- GitLab 17.0 or later, on GitLab.com or self-managed. From 17.0, GitLab runs a job's `after_script` when it cancels the job, which is how the pipeline records a cancel.
- An org on hushrun ([sign up](https://app.hushrun.com/sign-up)), and the project added to it: an org admin's `npx hushrun init` adds it.
- An org admin, to make the pipeline's agent token.
- The container registries your images already use, and a build that bakes in no per-environment configuration: see [When you need it](https://hushrun.com/docs/promotion.md#when).
- Linux runners. The install script supports Linux and macOS, and hushrun's container image is `linux/amd64` and `linux/arm64`.

## Set it up

1. Write the starter `hushrun.yaml`:

   ```bash
   hushrun init --pipeline \
     --staging-image us-docker.pkg.dev/acme-staging/api/server \
     --production-image us-docker.pkg.dev/acme-prod/api/server
   ```

   It recognizes a GitLab project by its `.gitlab-ci.yml`, or by its host, such as `gitlab.com` or `gitlab.example.com`.
2. Edit `hushrun.yaml`: the checks each stage requires and their commands, the scopes of a monorepo, `digest.exclude`, and the stages ([Promotion](https://hushrun.com/docs/promotion.md#stages)). Each scope covers `.gitlab-ci.yml` and anything it includes, so that a change to the pipeline changes the digest. Production finds staging's pipelines on staging's branch; when they run on another ref, name it on staging with `ci: {gitlab: {ref: <ref>}}`.
3. Run `npx hushrun init`. It adds `repository_url` and `transparency_log` to `hushrun.yaml`. Commit `hushrun.yaml`: hushrun reads it as committed, and the pipeline takes the repository's URL from it.
4. Make the pipeline's agent token, as an org admin, in the project:

   ```bash
   hushrun token create --agent gitlab-ci --expires 365d
   ```

   It prints the token, `hra_…`, and the agent's machine key, a PEM block, once: every agent token comes with its key. A token lasts 30 days unless `--expires` says otherwise, 365 days at most: make the next one under the same name before it expires.
5. Set the CI/CD variables ([below](#variables)).
6. Add the pipeline ([below](#pipeline)): `.gitlab-ci.yml` at the project's root, and `install-hushrun.sh` at `ci/install-hushrun.sh`.
7. Check the setup with `hushrun doctor` ([below](#doctor)).
8. Push to `develop`. Staging plans `full`: it runs the checks, builds, deploys, and records each step. Merge to `main`. Production plans `promote`: it copies staging's image to the production registry by digest and deploys it, with no build and no checks to repeat. The plan job's log shows the decision and why.

## CI/CD variables

In Settings, CI/CD, Variables:

| Variable | Type | Flags | Value |
|---|---|---|---|
| `HUSHRUN_TOKEN` | Variable | Masked, protected | The agent's token, `hra_…` |
| `HUSHRUN_MACHINE_KEY_FILE` | File | Protected | The agent's machine key: the whole PEM block |
| `HUSHRUN_GITLAB_TOKEN` | Variable | Masked | A project access token with the Reporter role and `read_api` |

- The key is a File variable, because GitLab can't mask a PEM key. GitLab writes it to a file outside the repository, and `HUSHRUN_MACHINE_KEY_FILE` holds the file's path. Every job signs with it, so the pipeline sets nothing for signing.
- The pipeline takes the repository's URL from `hushrun.yaml`, so it needs no `HUSHRUN_REPOSITORY_URL`. Set one only to pin the URL in the project's settings too: it must then match `hushrun.yaml`'s, or commands exit 2.
- On a hushrun server of your own, set `HUSHRUN_HOST` to its origin, such as `https://hushrun.example.com`, so that the agent's token goes there.
- With `HUSHRUN_GITLAB_TOKEN`, plans ask GitLab whether staging's pipelines are still running, or how they ended. Make it in Settings, Access tokens. Without it, a staging pipeline that ended without recording its outcome holds production until its lease runs out, 3 hours by default.
- Variables someone sets when they run a pipeline outrank project variables, so whoever may set them could point the pipeline at another server, or change how it signs. Set Settings, CI/CD, Variables, **Minimum role to use pipeline variables** to **No one allowed**, or to **Maintainer** if Maintainers force full builds. `hushrun doctor` reports the setting when its token may read it.

## The pipeline

Take `install-hushrun.sh` from the hushrun release you pin, and read it before you commit it: it installs that exact release for the runner's OS and architecture, after checking the archive's SHA-256 against the checksum pinned in it. `HUSHRUN_DOWNLOAD_URL` points it at a mirror of the release archives.

Then add this `.gitlab-ci.yml`, and set:

- `HUSHRUN_VERSION` and `HUSHRUN_IMAGE`, the same exact version, and the image's digest;
- the `test` job's image: the toolchain your checks need;
- the `deploy` job's image and scripts: `./build.sh "$HUSHRUN_REF"` pushes the image to `$HUSHRUN_REF`, and `./deploy.sh "$HUSHRUN_REF"` deploys it, pinned by digest;
- your cloud's credential step for the registry and the deploy, in `.hushrun` ([Cloud access](#cloud-access)).

```yaml title=".gitlab-ci.yml"
# Staging deploys develop; production deploys main with the image staging built and validated,
# copied by digest. Names, images and the build and deploy scripts are placeholders.
# The project's CI/CD variables, never set in this file: HUSHRUN_TOKEN, HUSHRUN_MACHINE_KEY_FILE
# and HUSHRUN_GITLAB_TOKEN (hushrun.com/docs/promotion/gitlab#variables).

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
      variables:
        HUSHRUN_STAGE: staging
    - if: '$CI_COMMIT_BRANCH == "main"'
      variables:
        HUSHRUN_STAGE: production
    - when: never

stages: [plan, test, deploy, finish, notify]

variables:
  HUSHRUN_VERSION: "0.1.0" # exact, as is the image tag below; never a moving tag
  HUSHRUN_IMAGE: ghcr.io/hushrun/hushrun:0.1.0 # pin its digest too: …:0.1.0@sha256:…

# Every job that runs hushrun extends this. GitLab's `extends` replaces a before_script rather
# than merging it, so your cloud's credential step, if it needs one, goes here too.
.hushrun:
  before_script:
    # The image has hushrun; on any other image, install the pinned release, checked against
    # its checksum.
    - '[ "$(hushrun --version 2>/dev/null)" = "hushrun $HUSHRUN_VERSION" ] || export PATH="$(dirname "$(sh ci/install-hushrun.sh)"):$PATH"'
  # A cancel records `superseded`: GitLab runs after_script, in a shell of its own, when it
  # cancels a running job. A timeout is a failure, which mark-failed records.
  after_script:
    - |
      if [ "$CI_JOB_STATUS" = canceled ] && [ -f plan.json ]; then
        case "$HUSHRUN_MODE" in
          promote | refresh | full)
            [ "$(hushrun --version 2>/dev/null)" = "hushrun $HUSHRUN_VERSION" ] || export PATH="$(dirname "$(sh ci/install-hushrun.sh)"):$PATH"
            hushrun mark superseded --stage "$HUSHRUN_STAGE" --plan plan.json --reason cancelled || true
            ;;
        esac
      fi

plan:
  stage: plan
  extends: .hushrun
  image: $HUSHRUN_IMAGE
  script:
    - hushrun plan --stage "$HUSHRUN_STAGE" --start --out plan.json --dotenv plan.env
  artifacts:
    paths: [plan.json]
    reports:
      dotenv: plan.env # HUSHRUN_MODE, HUSHRUN_CHECKS, HUSHRUN_DIGEST, HUSHRUN_REF, …

test:
  stage: test
  extends: .hushrun
  image: golang:1.25 # the toolchain the checks need
  needs: [plan]
  script:
    - '[ -n "$HUSHRUN_CHECKS" ] || { echo "nothing to run"; exit 0; }'
    - hushrun run --stage "$HUSHRUN_STAGE" --plan plan.json # exactly the plan's checks

deploy:
  stage: deploy
  extends: .hushrun
  image: google/cloud-sdk:slim # what build.sh and deploy.sh need
  needs: [plan, test]
  resource_group: deploy-$HUSHRUN_STAGE # one deploy at a time per stage
  script:
    - 'case "$HUSHRUN_MODE" in promote|refresh|full) ;; *) echo "mode $HUSHRUN_MODE: nothing to deploy"; exit 0 ;; esac'
    # Each output goes through a variable, so a command that fails fails the job.
    - out=$(hushrun guard --stage "$HUSHRUN_STAGE" --plan plan.json --format env); eval "$out"
    - '[ "$HUSHRUN_STALE" != true ] || { echo "a newer revision took over"; exit 0; }'
    - out=$(hushrun artifact resolve --stage "$HUSHRUN_STAGE" --plan plan.json --format env); eval "$out"
    - |
      if [ "$HUSHRUN_BUILD" = true ]; then
        ./build.sh "$HUSHRUN_REF"
        out=$(hushrun attest --stage "$HUSHRUN_STAGE" --plan plan.json --artifact "image=$HUSHRUN_REF" --format env)
        eval "$out"
      fi
    - ./deploy.sh "$HUSHRUN_REF"
    - hushrun mark deployed --stage "$HUSHRUN_STAGE" --plan plan.json --artifact "image=$HUSHRUN_REF"
    # Deployment checks run here: hushrun run e2e --stage "$HUSHRUN_STAGE" --plan plan.json
    - hushrun mark passed --stage "$HUSHRUN_STAGE" --plan plan.json --artifact "image=$HUSHRUN_REF"

mark-failed:
  stage: finish
  extends: .hushrun
  image: $HUSHRUN_IMAGE
  when: on_failure
  script:
    - 'case "$HUSHRUN_MODE" in promote|refresh|full) ;; *) exit 0 ;; esac' # nothing started
    - hushrun mark failed --stage "$HUSHRUN_STAGE" --plan plan.json --reason "pipeline failed"

# Staging only: plan production again once staging's outcome is recorded, after mark-failed.
plan-production:
  stage: notify
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
      when: always
  inherit:
    variables: false # forward nothing: staging's HUSHRUN_STAGE must not reach production
  trigger:
    project: $CI_PROJECT_PATH # this project; the downstream pipeline counts as an upstream trigger
    branch: main
```

What each job does:

- **plan** decides once. It records `started`, keeps the plan as the artifact `plan.json`, and passes its outputs to later jobs as a dotenv report: `HUSHRUN_MODE`, `HUSHRUN_CHECKS`, `HUSHRUN_DIGEST`, `HUSHRUN_REF` and more. The dotenv report leaves out the plan itself, which is why `plan.json` travels as an artifact. A `blocked` plan exits 3, which fails the pipeline on purpose.
- **test** runs exactly the plan's checks, and exits 8 when one fails.
- **deploy** runs `guard`, which stops a deploy of a revision that is no longer the branch's head; `artifact resolve`, which copies the upstream's image by digest, or asks for a build; the build and `attest` when the plan builds; your deploy; and `mark deployed` and `mark passed`. Deployment checks other stages require of this one run between the two marks.
- **mark-failed** records `failed` when a job failed or timed out. A job GitLab cancels records `superseded` from the `.hushrun` template's `after_script`.
- **plan-production**, on staging's pipelines only, starts a pipeline on `main` once staging's outcome is recorded, so production plans again. It forwards no variables, and it runs even when no one may set pipeline variables. Starting a pipeline on a protected `main` needs the right to run pipelines there.

Two rules hold in every job:

- A command whose output a job evaluates goes through a variable, `out=$(hushrun guard … --format env); eval "$out"`, because `eval "$(hushrun guard …)"` succeeds when hushrun fails.
- Every job carries out the plan for the pipeline's commit: never check out another ref, and never pass `--rev`.

Jobs that only run hushrun use the image `ghcr.io/hushrun/hushrun:<version>`: Alpine, git, CA certificates and hushrun's static binary, with no entrypoint. The other jobs install the same release with the script.

## Cloud access

hushrun's server needs no cloud credentials. Your registry and your deploy do:

- Give the jobs those roles through your cloud's workload identity federation with GitLab's ID tokens, with no long-lived cloud key.
- Bind the access to the project's id and a protected ref.
- Put the credential step in `.hushrun`'s `before_script`: a job's own `before_script` would replace the template's.

## Merge requests

In a merge request's pipeline, hushrun reads `hushrun.yaml` from the branch the merge request targets, which was reviewed, never from the change. Every command that reads or writes events needs that branch, fetched, in `--config-ref`, or it exits 2:

```bash
git fetch --depth=1 origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
hushrun plan --stage pr --config-ref "origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
```

- The agent's token and key are protected variables, so they don't reach a merge request pipeline from an unprotected branch. The pipeline on this page runs hushrun's jobs on `develop` and `main` only.
- A merge request pipeline has no branch, so it never stands for a deploy stage. Give its checks a stage of their own, such as a `pr` stage with no `branch`, and deploy only from the stage branches.

## Check the setup

From a job, or on your machine with `HUSHRUN_GITLAB_TOKEN` and `CI_PROJECT_PATH` set (and `CI_API_V4_URL` on a self-managed GitLab):

```bash
hushrun doctor
```

Besides what it checks everywhere ([GitHub Actions](https://hushrun.com/docs/promotion/github.md#doctor) lists it), `doctor` checks on GitLab that `HUSHRUN_GITLAB_TOKEN` can read the project, that the stage branches are protected, and who may set pipeline variables. In a job, it checks that the server lists the job's machine key for the agent. It exits 2 when a check fails.

## Day to day

- **A plan says `wait`.** Staging is still running this digest. Staging's `plan-production` job plans production again when staging is done: nothing to do.
- **A plan says `blocked`.** The reason names the pipeline and what to do, usually "fix forward, or force".
- **Forcing a full build.** With **Minimum role to use pipeline variables** set to **Maintainer**, a Maintainer runs the pipeline for `main` with `HUSHRUN_FORCE_FULL=true`. A forced run rebuilds and runs every source check, but never waives an approval, and a stage with `allow_force: false` refuses it.
- **Retrying a job.** GitLab keeps the pipeline's id, and hushrun counts the attempts: a retried plan job starts a new attempt.
- **A newer commit landed while deploying.** `guard` records `superseded`, and the deploy job stops. The newer pipeline deploys instead.
- **The token expires.** Make the next one under the same agent name, which comes with a new key, and replace `HUSHRUN_TOKEN` and `HUSHRUN_MACHINE_KEY_FILE`. Then revoke the old one with `hushrun token revoke hra_XXXXXX`, its prefix from `hushrun token list`. Events the old key signed before keep counting.
- **The org's plan is used up.** New check results aren't recorded, and `run` and `attest` say so once, with where to upgrade. Everything else is recorded, and deploys go on: see [Plans and limits](https://hushrun.com/docs/plans.md#at-the-limit).

Exit codes are the same as on GitHub: see [Exit codes](https://hushrun.com/docs/promotion/github.md#exit-codes).

## Limits

- **Child pipelines** record runs under their own pipeline id: run hushrun's jobs in the parent pipeline.
- **Merge request pipelines** are for checks, not stages ([above](#merge-requests)).
- **Windows runners**: the install script is for Linux and macOS.
- **`hushrun adopt`** writes GitHub workflows only: start a GitLab project from the pipeline on this page.
- **Keyless signing** with GitLab's ID tokens (`id_tokens: {SIGSTORE_ID_TOKEN: {aud: sigstore}}` and `HUSHRUN_SIGNING: sigstore`) counts as the project's CI only on a server that verifies GitLab's tokens for the project. hushrun's server doesn't, so sign with the agent's key there.