Skip to content
hushrunDocs

Promotion on GitLab CI

This page sets up a GitLab project that deploys staging from develop and production from main, so that production deploys the image staging built and validated, copied by digest. For what promotion is, start at Promotion.

How a GitLab pipeline signs

hushrun's server verifies GitHub Actions' ID tokens, not GitLab's. So a GitLab pipeline signs and writes as one of the org's agents, with two of the project's CI/CD variables:

  • The agent's token, HUSHRUN_TOKEN. hushrun sends it to the repository's URL that hushrun.yaml names, because that URL is on hushrun's host, which takes an agent token for its org only. Other tokens, such as HUSHRUN_REPOSITORY_TOKEN, never go to a URL that only a file in the repository names.
  • The machine key registered with the token, HUSHRUN_MACHINE_KEY_FILE. It signs every event, and the events count as agent:<name> at every stage that isn't CI-only, until an admin revokes the token. Events it signed before keep counting.

A CI-only stage counts only the repository's own CI, so don't make a GitLab project's stages CI-only: its pipeline's events wouldn't count there.

What you need

  • The Maintainer role on the project, with develop and main protected (Settings, Repository, Protected branches). Protected variables reach only protected branches, so the agent's token and key reach only your stage pipelines.
  • GitLab 17.0 or later, on GitLab.com or self-managed. From 17.0, GitLab runs a job's after_script when it cancels the job, which is how the pipeline records a cancel.
  • An org on hushrun (sign up), and the project added to it: an org admin's npx hushrun init adds it.
  • An org admin, to make the pipeline's agent token.
  • The container registries your images already use, and a build that bakes in no per-environment configuration: see When you need it.
  • Linux runners. The install script supports Linux and macOS, and hushrun's container image is linux/amd64 and linux/arm64.

Set it up

  1. Write the starter hushrun.yaml:

    hushrun init --pipeline \
  --staging-image us-docker.pkg.dev/acme-staging/api/server \
  --production-image us-docker.pkg.dev/acme-prod/api/server

    It recognizes a GitLab project by its .gitlab-ci.yml, or by its host, such as gitlab.com or gitlab.example.com.

  2. Edit hushrun.yaml: the checks each stage requires and their commands, the scopes of a monorepo, digest.exclude, and the stages (Promotion). Each scope covers .gitlab-ci.yml and anything it includes, so that a change to the pipeline changes the digest. Production finds staging's pipelines on staging's branch; when they run on another ref, name it on staging with ci: {gitlab: {ref: <ref>}}.

  3. Run npx hushrun init. It adds repository_url and transparency_log to hushrun.yaml. Commit hushrun.yaml: hushrun reads it as committed, and the pipeline takes the repository's URL from it.

  4. Make the pipeline's agent token, as an org admin, in the project:

    hushrun token create --agent gitlab-ci --expires 365d

    It prints the token, hra_…, and the agent's machine key, a PEM block, once: every agent token comes with its key. A token lasts 30 days unless --expires says otherwise, 365 days at most: make the next one under the same name before it expires.

  5. Set the CI/CD variables (below).

  6. Add the pipeline (below): .gitlab-ci.yml at the project's root, and install-hushrun.sh at ci/install-hushrun.sh.

  7. Check the setup with hushrun doctor (below).

  8. Push to develop. Staging plans full: it runs the checks, builds, deploys, and records each step. Merge to main. Production plans promote: it copies staging's image to the production registry by digest and deploys it, with no build and no checks to repeat. The plan job's log shows the decision and why.

CI/CD variables

In Settings, CI/CD, Variables:

VariableTypeFlagsValue
HUSHRUN_TOKENVariableMasked, protectedThe agent's token, hra_…
HUSHRUN_MACHINE_KEY_FILEFileProtectedThe agent's machine key: the whole PEM block
HUSHRUN_GITLAB_TOKENVariableMaskedA project access token with the Reporter role and read_api
  • The key is a File variable, because GitLab can't mask a PEM key. GitLab writes it to a file outside the repository, and HUSHRUN_MACHINE_KEY_FILE holds the file's path. Every job signs with it, so the pipeline sets nothing for signing.
  • The pipeline takes the repository's URL from hushrun.yaml, so it needs no HUSHRUN_REPOSITORY_URL. Set one only to pin the URL in the project's settings too: it must then match hushrun.yaml's, or commands exit 2.
  • On a hushrun server of your own, set HUSHRUN_HOST to its origin, such as https://hushrun.example.com, so that the agent's token goes there.
  • With HUSHRUN_GITLAB_TOKEN, plans ask GitLab whether staging's pipelines are still running, or how they ended. Make it in Settings, Access tokens. Without it, a staging pipeline that ended without recording its outcome holds production until its lease runs out, 3 hours by default.
  • Variables someone sets when they run a pipeline outrank project variables, so whoever may set them could point the pipeline at another server, or change how it signs. Set Settings, CI/CD, Variables, Minimum role to use pipeline variables to No one allowed, or to Maintainer if Maintainers force full builds. hushrun doctor reports the setting when its token may read it.

The pipeline

Take install-hushrun.sh from the hushrun release you pin, and read it before you commit it: it installs that exact release for the runner's OS and architecture, after checking the archive's SHA-256 against the checksum pinned in it. HUSHRUN_DOWNLOAD_URL points it at a mirror of the release archives.

Then add this .gitlab-ci.yml, and set:

  • HUSHRUN_VERSION and HUSHRUN_IMAGE, the same exact version, and the image's digest;
  • the test job's image: the toolchain your checks need;
  • the deploy job's image and scripts: ./build.sh "$HUSHRUN_REF" pushes the image to $HUSHRUN_REF, and ./deploy.sh "$HUSHRUN_REF" deploys it, pinned by digest;
  • your cloud's credential step for the registry and the deploy, in .hushrun (Cloud access).
.gitlab-ci.yml
# Staging deploys develop; production deploys main with the image staging built and validated,
# copied by digest. Names, images and the build and deploy scripts are placeholders.
# The project's CI/CD variables, never set in this file: HUSHRUN_TOKEN, HUSHRUN_MACHINE_KEY_FILE
# and HUSHRUN_GITLAB_TOKEN (hushrun.com/docs/promotion/gitlab#variables).

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
      variables:
        HUSHRUN_STAGE: staging
    - if: '$CI_COMMIT_BRANCH == "main"'
      variables:
        HUSHRUN_STAGE: production
    - when: never

stages: [plan, test, deploy, finish, notify]

variables:
  HUSHRUN_VERSION: "0.1.0" # exact, as is the image tag below; never a moving tag
  HUSHRUN_IMAGE: ghcr.io/hushrun/hushrun:0.1.0 # pin its digest too: …:0.1.0@sha256:…

# Every job that runs hushrun extends this. GitLab's `extends` replaces a before_script rather
# than merging it, so your cloud's credential step, if it needs one, goes here too.
.hushrun:
  before_script:
    # The image has hushrun; on any other image, install the pinned release, checked against
    # its checksum.
    - '[ "$(hushrun --version 2>/dev/null)" = "hushrun $HUSHRUN_VERSION" ] || export PATH="$(dirname "$(sh ci/install-hushrun.sh)"):$PATH"'
  # A cancel records `superseded`: GitLab runs after_script, in a shell of its own, when it
  # cancels a running job. A timeout is a failure, which mark-failed records.
  after_script:
    - |
      if [ "$CI_JOB_STATUS" = canceled ] && [ -f plan.json ]; then
        case "$HUSHRUN_MODE" in
          promote | refresh | full)
            [ "$(hushrun --version 2>/dev/null)" = "hushrun $HUSHRUN_VERSION" ] || export PATH="$(dirname "$(sh ci/install-hushrun.sh)"):$PATH"
            hushrun mark superseded --stage "$HUSHRUN_STAGE" --plan plan.json --reason cancelled || true
            ;;
        esac
      fi

plan:
  stage: plan
  extends: .hushrun
  image: $HUSHRUN_IMAGE
  script:
    - hushrun plan --stage "$HUSHRUN_STAGE" --start --out plan.json --dotenv plan.env
  artifacts:
    paths: [plan.json]
    reports:
      dotenv: plan.env # HUSHRUN_MODE, HUSHRUN_CHECKS, HUSHRUN_DIGEST, HUSHRUN_REF, …

test:
  stage: test
  extends: .hushrun
  image: golang:1.25 # the toolchain the checks need
  needs: [plan]
  script:
    - '[ -n "$HUSHRUN_CHECKS" ] || { echo "nothing to run"; exit 0; }'
    - hushrun run --stage "$HUSHRUN_STAGE" --plan plan.json # exactly the plan's checks

deploy:
  stage: deploy
  extends: .hushrun
  image: google/cloud-sdk:slim # what build.sh and deploy.sh need
  needs: [plan, test]
  resource_group: deploy-$HUSHRUN_STAGE # one deploy at a time per stage
  script:
    - 'case "$HUSHRUN_MODE" in promote|refresh|full) ;; *) echo "mode $HUSHRUN_MODE: nothing to deploy"; exit 0 ;; esac'
    # Each output goes through a variable, so a command that fails fails the job.
    - out=$(hushrun guard --stage "$HUSHRUN_STAGE" --plan plan.json --format env); eval "$out"
    - '[ "$HUSHRUN_STALE" != true ] || { echo "a newer revision took over"; exit 0; }'
    - out=$(hushrun artifact resolve --stage "$HUSHRUN_STAGE" --plan plan.json --format env); eval "$out"
    - |
      if [ "$HUSHRUN_BUILD" = true ]; then
        ./build.sh "$HUSHRUN_REF"
        out=$(hushrun attest --stage "$HUSHRUN_STAGE" --plan plan.json --artifact "image=$HUSHRUN_REF" --format env)
        eval "$out"
      fi
    - ./deploy.sh "$HUSHRUN_REF"
    - hushrun mark deployed --stage "$HUSHRUN_STAGE" --plan plan.json --artifact "image=$HUSHRUN_REF"
    # Deployment checks run here: hushrun run e2e --stage "$HUSHRUN_STAGE" --plan plan.json
    - hushrun mark passed --stage "$HUSHRUN_STAGE" --plan plan.json --artifact "image=$HUSHRUN_REF"

mark-failed:
  stage: finish
  extends: .hushrun
  image: $HUSHRUN_IMAGE
  when: on_failure
  script:
    - 'case "$HUSHRUN_MODE" in promote|refresh|full) ;; *) exit 0 ;; esac' # nothing started
    - hushrun mark failed --stage "$HUSHRUN_STAGE" --plan plan.json --reason "pipeline failed"

# Staging only: plan production again once staging's outcome is recorded, after mark-failed.
plan-production:
  stage: notify
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
      when: always
  inherit:
    variables: false # forward nothing: staging's HUSHRUN_STAGE must not reach production
  trigger:
    project: $CI_PROJECT_PATH # this project; the downstream pipeline counts as an upstream trigger
    branch: main

What each job does:

  • plan decides once. It records started, keeps the plan as the artifact plan.json, and passes its outputs to later jobs as a dotenv report: HUSHRUN_MODE, HUSHRUN_CHECKS, HUSHRUN_DIGEST, HUSHRUN_REF and more. The dotenv report leaves out the plan itself, which is why plan.json travels as an artifact. A blocked plan exits 3, which fails the pipeline on purpose.
  • test runs exactly the plan's checks, and exits 8 when one fails.
  • deploy runs guard, which stops a deploy of a revision that is no longer the branch's head; artifact resolve, which copies the upstream's image by digest, or asks for a build; the build and attest when the plan builds; your deploy; and mark deployed and mark passed. Deployment checks other stages require of this one run between the two marks.
  • mark-failed records failed when a job failed or timed out. A job GitLab cancels records superseded from the .hushrun template's after_script.
  • plan-production, on staging's pipelines only, starts a pipeline on main once staging's outcome is recorded, so production plans again. It forwards no variables, and it runs even when no one may set pipeline variables. Starting a pipeline on a protected main needs the right to run pipelines there.

Two rules hold in every job:

  • A command whose output a job evaluates goes through a variable, out=$(hushrun guard … --format env); eval "$out", because eval "$(hushrun guard …)" succeeds when hushrun fails.
  • Every job carries out the plan for the pipeline's commit: never check out another ref, and never pass --rev.

Jobs that only run hushrun use the image ghcr.io/hushrun/hushrun:<version>: Alpine, git, CA certificates and hushrun's static binary, with no entrypoint. The other jobs install the same release with the script.

Cloud access

hushrun's server needs no cloud credentials. Your registry and your deploy do:

  • Give the jobs those roles through your cloud's workload identity federation with GitLab's ID tokens, with no long-lived cloud key.
  • Bind the access to the project's id and a protected ref.
  • Put the credential step in .hushrun's before_script: a job's own before_script would replace the template's.

Merge requests

In a merge request's pipeline, hushrun reads hushrun.yaml from the branch the merge request targets, which was reviewed, never from the change. Every command that reads or writes events needs that branch, fetched, in --config-ref, or it exits 2:

git fetch --depth=1 origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
hushrun plan --stage pr --config-ref "origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
  • The agent's token and key are protected variables, so they don't reach a merge request pipeline from an unprotected branch. The pipeline on this page runs hushrun's jobs on develop and main only.
  • A merge request pipeline has no branch, so it never stands for a deploy stage. Give its checks a stage of their own, such as a pr stage with no branch, and deploy only from the stage branches.

Check the setup

From a job, or on your machine with HUSHRUN_GITLAB_TOKEN and CI_PROJECT_PATH set (and CI_API_V4_URL on a self-managed GitLab):

hushrun doctor

Besides what it checks everywhere (GitHub Actions lists it), doctor checks on GitLab that HUSHRUN_GITLAB_TOKEN can read the project, that the stage branches are protected, and who may set pipeline variables. In a job, it checks that the server lists the job's machine key for the agent. It exits 2 when a check fails.

Day to day

  • A plan says wait. Staging is still running this digest. Staging's plan-production job plans production again when staging is done: nothing to do.
  • A plan says blocked. The reason names the pipeline and what to do, usually "fix forward, or force".
  • Forcing a full build. With Minimum role to use pipeline variables set to Maintainer, a Maintainer runs the pipeline for main with HUSHRUN_FORCE_FULL=true. A forced run rebuilds and runs every source check, but never waives an approval, and a stage with allow_force: false refuses it.
  • Retrying a job. GitLab keeps the pipeline's id, and hushrun counts the attempts: a retried plan job starts a new attempt.
  • A newer commit landed while deploying. guard records superseded, and the deploy job stops. The newer pipeline deploys instead.
  • The token expires. Make the next one under the same agent name, which comes with a new key, and replace HUSHRUN_TOKEN and HUSHRUN_MACHINE_KEY_FILE. Then revoke the old one with hushrun token revoke hra_XXXXXX, its prefix from hushrun token list. Events the old key signed before keep counting.
  • The org's plan is used up. New check results aren't recorded, and run and attest say so once, with where to upgrade. Everything else is recorded, and deploys go on: see Plans and limits.

Exit codes are the same as on GitHub: see Exit codes.

Limits

  • Child pipelines record runs under their own pipeline id: run hushrun's jobs in the parent pipeline.
  • Merge request pipelines are for checks, not stages (above).
  • Windows runners: the install script is for Linux and macOS.
  • hushrun adopt writes GitHub workflows only: start a GitLab project from the pipeline on this page.
  • Keyless signing with GitLab's ID tokens (id_tokens: {SIGSTORE_ID_TOKEN: {aud: sigstore}} and HUSHRUN_SIGNING: sigstore) counts as the project's CI only on a server that verifies GitLab's tokens for the project. hushrun's server doesn't, so sign with the agent's key there.
Markdown