Skip to content
Docs

Deploys on GitLab CI

Updated

This guide sets up a GitLab project that deploys from main through hushrun's pipeline. A push of files that already passed deploys the image built for them, by digest, with no build and no check to repeat. For what a deploy pipeline is, start at Deploys.

Before you begin

  • The Maintainer role on the project, with main protected (Settings, Repository, Protected branches). Protected variables reach only protected branches, so the token and key reach only your deploy pipelines.
  • GitLab 17.0 or later, on GitLab.com or self-managed. It runs a job's after_script when it cancels the job, which is how the pipeline records a cancel.
  • An org on hushrun (sign up), with the project added: an org admin's npx hushrun init adds it.
  • An org admin, to make the pipeline's agent token.
  • The container registry your image already uses.
  • Linux runners.

A GitLab pipeline signs as one of the org's agents, not as the project's CI. See How a GitLab pipeline signs.

Write hushrun.yaml

hushrun init --pipeline --image us-docker.pkg.dev/acme/api/server

It recognizes a GitLab project by its .gitlab-ci.yml, or by its host, such as gitlab.com or gitlab.example.com. Edit the file: the checks the deploy requires and their commands, the branch, and in a monorepo the scopes and digest.exclude (What a deploy requires).

Each scope covers .gitlab-ci.yml and anything it includes, so that a change to the pipeline changes the digest.

Connect the project

Run npx hushrun init. It adds repository_url and transparency_log to hushrun.yaml. Commit the file: hushrun reads it as committed, and the pipeline takes the repository's URL from it.

Make the pipeline's agent token

As an org admin, in the project, at a terminal:

hushrun token create --agent gitlab-ci --expires 365d

It prints the token, hra_…, and the agent's machine key, a PEM block, once, and keeps the key out of anything that could be a log. In a script, --format json gives them. The HUSHRUN_AGENT string it also shows is for sandboxes, not pipelines.

A token lasts 30 days unless --expires says otherwise, 365 days at most. Make the next one before it expires (Day to day).

Set the CI/CD variables

In Settings, CI/CD, Variables:

VariableTypeFlagsValue
HUSHRUN_TOKENVariableMasked, protectedThe agent's token, hra_…
HUSHRUN_MACHINE_KEY_FILEFileProtectedThe agent's machine key: the whole PEM block
HUSHRUN_GITLAB_TOKENVariableMaskedOptional: a project access token with read_api, for hushrun doctor

On the same page, set Minimum role to use pipeline variables to No one allowed, or to Maintainer if Maintainers force full runs. Variables set when someone runs a pipeline outrank project variables, so whoever may set them could point the pipeline at another server, or change how it signs.

  • The key is a File variable because GitLab can't mask a PEM key. GitLab writes it to a file outside the repository, and HUSHRUN_MACHINE_KEY_FILE holds the file's path. Every job signs with it.
  • With HUSHRUN_GITLAB_TOKEN, hushrun doctor checks that main is protected, and who may set pipeline variables. Nothing else uses it: plans never ask GitLab about pipelines. Make it in Settings, Access tokens. Reading who may set pipeline variables takes the Maintainer role.
  • The pipeline needs no HUSHRUN_REPOSITORY_URL. Set one only to pin the URL in the project's settings too: it must then match hushrun.yaml's, or commands exit 2.
  • On a hushrun server of your own, set HUSHRUN_HOST to its origin, such as https://hushrun.example.com, so that the agent's token goes there.

Add the pipeline

  1. Take install-hushrun.sh from a hushrun release, read it, and commit it as ci/install-hushrun.sh. It installs a release for the runner's OS and architecture, after checking the archive's SHA-256 (details).
  2. Add this .gitlab-ci.yml at the project's root, and set:
    • the test job's image: the toolchain your checks need;
    • the deploy job's image and scripts: ./build.sh "$HUSHRUN_REF" pushes the image to $HUSHRUN_REF, and ./deploy.sh "$HUSHRUN_REF" deploys it, pinned by digest;
    • your cloud's credential step for the registry and the deploy, in .hushrun (Cloud access).
.gitlab-ci.yml
# Names, images and the build and deploy scripts are placeholders. The CI/CD variables are
# the project's, never set here (hushrun.com/docs/deploy/gitlab#variables).

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
    - when: never

stages: [plan, test, deploy, finish]

# The newest hushrun release: the plan job runs the image's, and every later job the same one,
# even when a newer one comes out meanwhile. To install one release until you change it, set
# HUSHRUN_VERSION, and the image to that release, pinned by the digest its release notes give:
#   HUSHRUN_VERSION: "0.9.0"
#   HUSHRUN_IMAGE: ghcr.io/hushrun/hushrun:0.9.0@sha256:…
variables:
  HUSHRUN_IMAGE: ghcr.io/hushrun/hushrun:latest

# Every job that runs hushrun extends this. GitLab's `extends` replaces a before_script rather
# than merging it, so your cloud's credential step for the registry, if it needs one, goes
# here too (https://hushrun.com/docs/deploy/gitlab#cloud-access).
.hushrun:
  before_script:
    # The image has hushrun. On any other image, or when it isn't HUSHRUN_VERSION, the release
    # the plan job ran (from its dotenv report), install that release, checked against its
    # checksum.
    - |
      have=$(hushrun --version 2>/dev/null || true)
      if [ -z "$have" ] || { [ -n "${HUSHRUN_VERSION:-}" ] && [ "$have" != "hushrun $HUSHRUN_VERSION" ]; }; then
        bin=$(sh ci/install-hushrun.sh)
        PATH="$(dirname "$bin"):$PATH"
      fi
  # A cancel records `superseded`: GitLab runs after_script, in a shell of its own, when it
  # cancels a running job. A timeout is a failure, which mark-failed records.
  after_script:
    - |
      if [ "$CI_JOB_STATUS" = canceled ] && [ -f plan.json ]; then
        case "$HUSHRUN_MODE" in
          promote | refresh | full)
            have=$(hushrun --version 2>/dev/null || true)
            if [ -z "$have" ] || { [ -n "${HUSHRUN_VERSION:-}" ] && [ "$have" != "hushrun $HUSHRUN_VERSION" ]; }; then
              bin=$(sh ci/install-hushrun.sh)
              PATH="$(dirname "$bin"):$PATH"
            fi
            hushrun mark superseded --plan plan.json --reason cancelled || true
            ;;
        esac
      fi

plan:
  stage: plan
  extends: .hushrun
  image: $HUSHRUN_IMAGE
  script:
    - hushrun plan --start --out plan.json --dotenv plan.env
    # Later jobs run this release, the one that made the plan.
    - echo "HUSHRUN_VERSION=$(hushrun --version | cut -d ' ' -f 2)" >> plan.env
  artifacts:
    paths: [plan.json]
    reports:
      dotenv: plan.env # HUSHRUN_MODE, HUSHRUN_CHECKS, HUSHRUN_DIGEST, HUSHRUN_REF, …, HUSHRUN_VERSION

test:
  stage: test
  extends: .hushrun
  image: golang:1.25 # the toolchain the checks need
  needs: [plan]
  script:
    - '[ -n "$HUSHRUN_CHECKS" ] || { echo "nothing to run"; exit 0; }'
    - hushrun run --plan plan.json # exactly the plan's checks

deploy:
  stage: deploy
  extends: .hushrun
  image: google/cloud-sdk:slim # what build.sh and deploy.sh need
  needs: [plan, test]
  resource_group: deploy # one deploy at a time
  script:
    - 'case "$HUSHRUN_MODE" in promote|refresh|full) ;; *) echo "mode $HUSHRUN_MODE: nothing to deploy"; exit 0 ;; esac'
    # Each output goes through a variable, so a command that fails fails the job.
    - out=$(hushrun guard --plan plan.json --format env); eval "$out"
    - '[ "$HUSHRUN_STALE" != true ] || { echo "a newer revision took over"; exit 0; }'
    - out=$(hushrun artifact resolve --plan plan.json --format env); eval "$out"
    - |
      if [ "$HUSHRUN_BUILD" = true ]; then
        ./build.sh "$HUSHRUN_REF"
        out=$(hushrun attest --plan plan.json --artifact "image=$HUSHRUN_REF" --format env)
        eval "$out"
      fi
    - ./deploy.sh "$HUSHRUN_REF"
    - hushrun mark deployed --plan plan.json --artifact "image=$HUSHRUN_REF"
    # A check of what was just deployed, such as a smoke test, runs here: its failure fails the
    # job, and mark-failed records it.
    - hushrun mark passed --plan plan.json --artifact "image=$HUSHRUN_REF"

mark-failed:
  stage: finish
  extends: .hushrun
  image: $HUSHRUN_IMAGE
  when: on_failure
  script:
    - 'case "$HUSHRUN_MODE" in promote|refresh|full) ;; *) exit 0 ;; esac' # nothing started
    - hushrun mark failed --plan plan.json --reason "pipeline failed"

The pipeline's jobs explains each job.

Check the setup

From a job, or on your machine with HUSHRUN_GITLAB_TOKEN and CI_PROJECT_PATH set (and CI_API_V4_URL on a self-managed GitLab):

hushrun doctor

It exits 2 when a check fails. Besides what it checks everywhere (Commands), on GitLab it checks that HUSHRUN_GITLAB_TOKEN can read the project, that main is protected, and who may set pipeline variables. In a job, it checks that the server lists the job's machine key for the agent.

Push to main

Push to main. The first pipeline plans full: it runs the checks, builds, deploys, and records each step. Run a new pipeline for the same files, or later revert to them, and it plans promote: it deploys the image it already built and checked, by digest, with no build and no check to repeat. The plan job's log starts with the mode and the reason.

Day to day

  • blocked. The pipeline fails on purpose, and the plan job's log says what to do: the pipeline runs on another branch than deploy.branch, or it was forced where allow_force: false forbids it.
  • current. The project already runs this code. To deploy it again, run a new pipeline for main by hand. Retrying the plan job doesn't: a current plan records nothing, so the retry is still a first attempt.
  • refresh. Only the checks without valid evidence run (Which checks run), and the image is still reused.
  • Force a full run. A Maintainer runs the pipeline for main with HUSHRUN_FORCE_FULL=true, which needs Minimum role to use pipeline variables set to Maintainer. A forced run builds and runs every required check.
  • Retry the plan job. GitLab keeps the pipeline's id, so hushrun plan counts attempts itself: one more than the pipeline recorded. The other jobs take the plan's attempt from plan.json.
  • A newer commit landed while deploying. guard records superseded, and the deploy job stops. The newer pipeline deploys instead.
  • Renew the token. Make the next one under the same agent name, which comes with a new key, and replace HUSHRUN_TOKEN and HUSHRUN_MACHINE_KEY_FILE. Then revoke the old one with hushrun token revoke hra_XXXXXX, its prefix from hushrun token list. What the old key signed keeps counting.
  • The org's plan is used up. New check results aren't recorded, and run and attest say so once, with where to upgrade. Everything else is recorded, and deploys go on: see Plans and limits.

Exit codes are the same as on GitHub: see Exit codes.

How a GitLab pipeline signs

hushrun's server verifies GitHub Actions' ID tokens, not GitLab's. So a GitLab pipeline signs and writes as one of the org's agents, with two CI/CD variables:

  • The agent's token, HUSHRUN_TOKEN. hushrun sends it to the repository's URL that hushrun.yaml names, because that URL is on hushrun's host, which takes an agent token for its org only. Other tokens, such as HUSHRUN_REPOSITORY_TOKEN, never go to a URL that only a file in the repository names.
  • The machine key that came with the token, HUSHRUN_MACHINE_KEY_FILE. It signs every event, which counts as agent:<name> until an admin revokes the token. What it signed before keeps counting.

CI reuses only CI's results counts only the repository's own CI, so don't turn it on for a GitLab project: its pipeline's events wouldn't count.

The pipeline's jobs

  • plan decides once. It records started, and passes its outputs to later jobs as a dotenv report: HUSHRUN_MODE, HUSHRUN_CHECKS, HUSHRUN_DIGEST, HUSHRUN_REF and more, and the release it ran, HUSHRUN_VERSION. The plan itself, which the report leaves out, travels as the artifact plan.json. A blocked plan exits 3, which fails the pipeline on purpose.
  • test runs exactly the plan's checks, and exits 8 when one fails.
  • deploy runs only when the mode is promote, refresh or full:
    1. guard stops a deploy of a revision that is no longer the branch's head.
    2. artifact resolve gives the image to deploy, pinned by digest, or asks for a build.
    3. When the plan builds, your build runs, and attest records the image.
    4. Your deploy runs, then mark deployed. A check of what was just deployed, such as a smoke test, goes here.
    5. mark passed.
  • mark-failed records failed when a job failed or timed out. A job GitLab cancels records superseded from the .hushrun template's after_script.

Two rules hold in every job:

  • A command whose output a job evaluates goes through a variable, out=$(hushrun guard … --format env); eval "$out", because eval "$(hushrun guard …)" succeeds when hushrun fails.
  • Every job carries out the plan for the pipeline's commit: never check out another ref, and never pass --rev.

Which hushrun the pipeline runs

Jobs that only run hushrun use the image ghcr.io/hushrun/hushrun: Alpine, git, CA certificates and hushrun's static binary, with no entrypoint. Its latest tag is the newest release, and each release has its version's tag.

The plan job runs latest and passes its version to the later jobs, which install that same release with install-hushrun.sh. A new release reaches your pipelines at their next run: its fixes without an edit, and a bad release as fast.

  • To pin a release, set HUSHRUN_VERSION, such as "0.9.0", and HUSHRUN_IMAGE to that release, pinned by the digest its release notes give. A pinned version changes only through a commit you review.
  • The script checks the archive against the checksum it pins for that release. For a release it doesn't list, such as a newer one, it uses the checksum published with the release.
  • Runners that can't reach GitHub need a pinned version. Mirror the image and the release archives where they can read them, and point HUSHRUN_IMAGE and HUSHRUN_DOWNLOAD_URL there.

Cloud access

hushrun's server needs no cloud credentials. Your registry and your deploy do:

  • Give the jobs those roles through your cloud's workload identity federation with GitLab's ID tokens, with no long-lived cloud key.
  • Bind the access to the project's id and a protected ref.
  • Put the credential step in .hushrun's before_script: a job's own before_script would replace the template's.

Merge requests

In a merge request's pipeline, hushrun reads hushrun.yaml from the branch the merge request targets, which was reviewed, never from the change. Fetch that branch and name it with --config-ref, or every command that reads or writes events exits 2:

git fetch --depth=1 origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
hushrun run --all --config-ref "origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
  • The token and key are protected variables, so they don't reach a merge request pipeline from an unprotected branch. The pipeline on this page runs hushrun's jobs on main only.
  • A merge request pipeline has no branch, so it never deploys. The checks it passes count for the same digest: when main hasn't moved, the merged commit has the same files, and the deploy doesn't run them again.

Limits

  • Child pipelines record runs under their own pipeline id: run hushrun's jobs in the parent pipeline.
  • Windows runners: the install script is for Linux and macOS.
  • hushrun adopt writes GitHub workflows only: start a GitLab project from the pipeline on this page.
  • Keyless signing with GitLab's ID tokens (id_tokens: {SIGSTORE_ID_TOKEN: {aud: sigstore}} and HUSHRUN_SIGNING: sigstore) counts as the project's CI only on a server that verifies GitLab's tokens for the project. hushrun's server doesn't, so sign with the agent's key there.

Next

Markdown