codician/diffinerypublic

---

#title: Runners description: Where Diffinery workflow jobs run: hosted Linux runners in Cloudflare Containers, and self-hosted runners you start with diffinery runner.

A runner is the machine that executes one workflow job. Diffinery offers two kinds: hosted runners, which run each job in a fresh Ubuntu 24.04 container on Cloudflare, and self-hosted runners, which run jobs on a machine you control (a Mac, a GPU box, a server on a private network) using the diffinery runner command. You pick one per job with runs-on, the same key GitHub Actions uses.

#How runs-on picks a runner

Diffinery reads the job's runs-on labels (case-insensitive) and decides:

1. If any label is self-hosted, the job waits for a self-hosted runner that offers every label in runs-on. See Self-hosted runners.
2. Otherwise, if every label is one Diffinery hosts, the job runs on a hosted runner.
3. Otherwise the job fails at once with:

No Diffinery runner matches runs-on: windows-latest. Linux (ubuntu-*) runners are hosted; anything else needs runs-on: [self-hosted, ...] and a registered runner.

Things to know:

• Every ubuntu-* label gets the same Ubuntu 24.04 x64 image, including ubuntu-22.04 and labels that ask for Arm, such as ubuntu-24.04-arm.
• runs-on can be a string, a list, or a mapping with labels. A group in the mapping form is ignored; only labels count.
• runs-on is evaluated for each matrix combination, so an entry can be an expression using github, inputs, vars, needs, matrix and strategy. An expression can give one label or a list of labels. Empty and null values are dropped.
• If evaluating runs-on fails, only that matrix combination fails, with Error evaluating the job for this matrix combination: <message>. If no labels are left, the job fails with No Diffinery runner matches runs-on: (no labels). ....

jobs:
  test:
    runs-on: ubuntu-latest        # hosted
    steps:
      - run: echo "hosted"
  build:
    strategy:
      matrix:
        target: [linux, macos]
    # The linux leg runs on a hosted runner; the macos leg waits for a
    # self-hosted runner with the macos label.
    runs-on: ${{ matrix.target == 'macos' && fromJSON('["self-hosted","macos"]') || 'ubuntu-latest' }}
    steps:
      - run: uname -a
  gpu:
    runs-on: [self-hosted, linux, gpu]   # waits for a runner started with --labels linux,gpu
    steps:
      - run: nvidia-smi

#Hosted runners

Each job on a hosted runner gets its own container on Cloudflare Containers, started for that job and thrown away when it finishes. Nothing carries over between jobs except what you save with caches and artifacts (see Caching and artifacts).

#Machine

#The Ubuntu 24.04 image

The hosted image is built from containers/runner/Dockerfile in the Diffinery source. It is deliberately small. This is the complete list of what is preinstalled:

Image settings that affect your steps:

• git config --system safe.directory '*' is set, and Git LFS is installed system-wide.
• PIP_BREAK_SYSTEM_PACKAGES=1 is set, so pip install works without a virtual environment, as on GitHub's hosted runners.
• COREPACK_ENABLE_DOWNLOAD_PROMPT=0 is set, so corepack never stops to ask.
• ImageOS=ubuntu24 and ImageVersion=20250101.1 are set, as on GitHub's ubuntu-24.04 image. actions/setup-go and actions/setup-python include them in their cache keys.
• zstd is installed, so actions/cache compresses with zstd, as on GitHub.
• The apt package lists are removed to keep the image small. Run apt-get update before apt-get install.

Anything else (Go, Java, Ruby, Rust, .NET, PowerShell, browsers, cloud CLIs) is not preinstalled. Install it in a step:

steps:
  - uses: actions/setup-go@v5          # downloads Go from go.dev
    with:
      go-version: "1.23"
  - run: apt-get update && apt-get install -y --no-install-recommends openjdk-17-jdk-headless
  - run: pip install awscli

actions/setup-node, actions/setup-python and actions/setup-go use the preinstalled Node.js or Python when its version satisfies the one you ask for, and download the version otherwise. See Compatibility with GitHub Actions for their supported inputs.

Note: Jobs run as root, and sudo runs commands without asking for a password, as on GitHub's runners. The image ships without apt package lists, so run sudo apt-get update before sudo apt-get install ....

#Docker on hosted runners

Docker works on hosted runners the same way as on GitHub's: job containers, service containers, Docker container actions, and docker build, docker run and docker compose in run: steps. Details:

• Started on first use. The docker command is a wrapper that starts the Docker daemon the first time anything in the job needs it (a docker command in a step, a job container:, services:, or a Docker action). The first use prints Starting the Docker daemon and takes a few seconds; jobs that never touch Docker don't pay for it. If the daemon can't start, the step fails with The Docker daemon did not start: followed by the end of the daemon's log.
• Docker Hub through mirror.gcr.io. The daemon uses https://mirror.gcr.io as its registry mirror, so pulls of Docker Hub images (redis:7-alpine, postgres:16) usually come from Google's public mirror and avoid Docker Hub's anonymous rate limits. Images the mirror doesn't have are pulled from Docker Hub as usual.
• DNS. The daemon gives containers the DNS servers 1.1.1.1 and 8.8.8.8.
• Pushing to the Diffinery registry. The daemon and BuildKit builders reach diffinery.com through a local relay that splits image layer uploads into chunks, so layers larger than Cloudflare's 100 MB request limit push from hosted workflows. See Container registry.
• Buildx builders. docker buildx create with the docker-container driver (which docker/setup-buildx-action runs) is handled by the wrapper: it starts the BuildKit container on the host network and registers it as a remote builder, because the normal driver can't work inside a Cloudflare Container. docker buildx rm removes it.

The full guide is Docker in workflows.

#Environment variables

Hosted and self-hosted runners set the GitHub-compatible variables (CI=true, GITHUB_ACTIONS=true, GITHUB_REPOSITORY, GITHUB_SHA, GITHUB_REF, GITHUB_WORKSPACE, GITHUB_ENV, GITHUB_OUTPUT, and so on) plus DIFFINERY=true. GITHUB_SERVER_URL is https://diffinery.com and GITHUB_API_URL is https://diffinery.com/api/v3. RUNNER_NAME is diffinery and RUNNER_ENVIRONMENT is diffinery-hosted on every runner. Workflows can override CI and DIFFINERY, but not GITHUB_* or RUNNER_* variables.

Other variables a job gets:

• ImageOS and ImageVersion, on hosted and self-hosted runners alike (ubuntu24 and 20250101.1 unless the runner's environment sets them).
• RUNNER_DEBUG=1 when the secret or variable ACTIONS_STEP_DEBUG or ACTIONS_RUNNER_DEBUG is true. See Expressions and contexts.
• ACTIONS_RUNTIME_TOKEN, ACTIONS_RUNTIME_URL, ACTIONS_RESULTS_URL, ACTIONS_CACHE_URL and ACTIONS_CACHE_SERVICE_V2, but only in JavaScript actions, Docker container actions and the built-in actions, never in run: steps. They let actions such as actions/cache and actions/upload-artifact use the cache and artifact services. See Caching and artifacts.

#Limits and timeouts

If the container stops unexpectedly (for example it runs out of memory), the job fails with The runner stopped unexpectedly (exit code N). A container that stops before its runner fetched the job is replaced instead, since no step ran.

The log's first lines name where the job ran, for example Runner: image diffinery-runner:96dbbe31790f, in fra06, EEUR, container 6f1c…. The jobs API returns the same as runner (image, location, instance), next to runner_name. GET /meta names the image the last deploy set as runner_image. While a deploy's rollout is updating instances, a new container can still start on the previous image. Diffinery turns that runner away before it runs anything and starts the job again on another container (after 30 s, then 1, 2 and 4 minutes). After four tries, the job runs on whatever image it got rather than wait for a rollout that has stalled. The jobs API's placementTries counts the times a job went back in line before a runner took it, and restarts counts the times it started again because the platform replaced its runner.

#Windows and macOS

Cloudflare Containers run Linux only, so Diffinery has no Windows or macOS hosted runners. runs-on: windows-latest or runs-on: macos-latest fails with the "No Diffinery runner matches runs-on" message. For macOS builds, Windows-specific tooling or GPUs, use a self-hosted runner on your own machine with runs-on: [self-hosted, ...].

#Self-hosted runners

A self-hosted runner is the diffinery runner command running on a machine you control. It polls one or more repositories for waiting jobs whose runs-on includes self-hosted, claims one, runs it with the Diffinery runner agent (the same program that runs inside hosted containers), streams the logs to Diffinery, and reports the result. Use it for:

• macOS: build and sign iOS or macOS apps with Xcode on a Mac.
• GPUs: run CUDA tests or model evaluation on a machine with a GPU.
• Private networks: deploy to or test against servers that Cloudflare can't reach (a database behind a VPN, an on-premises cluster).
• Long-lived caches and big toolchains: the runner's work directory and tool cache persist between jobs.

#Requirements

• Node.js 22 or later on the machine.

• The diffinery CLI. Install it with:

  curl -fsSL https://diffinery.com/-/cli/diffinery.mjs -o ~/.local/bin/diffinery && chmod +x ~/.local/bin/diffinery

• The runner agent, a single Node.js file named agent.mjs. Build it from a checkout of the Diffinery source:

  pnpm install
  pnpm --filter @diffinery/runner build      # produces containers/runner/dist/agent.mjs

  Copy containers/runner/dist/agent.mjs to the runner machine (for example to /opt/diffinery/agent.mjs) and pass it with --agent or the DIFFINERY_RUNNER_AGENT environment variable.

• A registration token for each repository the runner serves, from a person with the admin role on it. A registration token works for an hour, and the runner trades it for its own runner token, which can claim that repository's jobs and do nothing else. Agents, job tokens and people without admin can't create one (403 only repository admins can add self-hosted runners).

• Bash (the default shell for run: steps; sh is used when bash is missing), git and tar, which the built-in actions use.

• Docker, only if jobs use container:, services: or Docker actions.

#Start a runner

1. On your own computer, as a repository admin, create a registration token (it works for an hour):

   diffinery api POST /repos/acme/website/actions/runners/registration-token

   {"token": "drg_…", "expires_at": "2026-10-13T19:45:00.000Z"}

2. On the runner machine, register and start the runner. Set DIFFINERY_URL if you don't use https://diffinery.com. No login is needed:

   diffinery runner -R acme/website --registration-token drg_… --labels macos,arm64 --agent /opt/diffinery/agent.mjs

   It registers, saves the runner's credentials, prints the labels it offers and waits:

   Registered runner my-mac with acme/website (saved to ~/.config/diffinery/runners/diffinery.com/acme/website/my-mac.json)
   Runner my-mac offering [self-hosted, macos, arm64] for acme/website
   Waiting for jobs. Press Ctrl-C to stop.

   Later starts use the saved registration: diffinery runner -R acme/website --agent /opt/diffinery/agent.mjs. Its labels are the ones it registered with. To change them, delete the saved file and register again.

   On a machine where you're signed in as a repository admin (diffinery login), diffinery runner registers itself with your token and claims jobs with the runner's own token. Prefer a registration token on shared or dedicated runner machines, so no personal token is kept where jobs could read it.

3. Target it from a workflow:

   jobs:
     ios:
       runs-on: [self-hosted, macos, arm64]
       steps:
         - uses: actions/checkout@v4
         - run: xcodebuild -scheme App -destination 'platform=iOS Simulator,name=iPhone 16' test

When a job arrives the runner prints → acme/website: running job ios (run <run id>, job <job id>), echoes the step output prefixed with the job name, and ends with ✓ job ios finished or X job ios finished with exit code 1.

#diffinery runner options

The full generated reference is at CLI reference.

#Labels and job matching

• A job is eligible when its runs-on contains self-hosted and every one of its labels is offered by the runner. The runner offers self-hosted plus everything in --labels. Comparison is case-insensitive.
• A runner with more labels than a job asks for can still take the job: --labels linux,gpu,cuda12 takes runs-on: [self-hosted, gpu].
• runs-on: [self-hosted] alone can be taken by any runner serving the repository.
• When several jobs are waiting, a runner takes the oldest run's jobs first.
• Labels have no meaning to Diffinery beyond matching. Choose names that describe the machine (macos, arm64, gpu, vpn-prod).

#How jobs are claimed

1. When a self-hosted job is ready to start, Diffinery marks it waiting and creates a queued check on the commit.

2. Each runner polls POST /api/v3/repos/{owner}/{repo}/actions/runners/claim with its runner token, every --interval seconds per repository while idle. It offers the labels it registered with. The response is 204 No Content when nothing matches, or the job with a fresh job token:

   POST /api/v3/repos/acme/website/actions/runners/claim
   Authorization: Bearer drn_...
   Content-Type: application/json
   
   {}

   {"job_id": "…", "run_id": "…", "name": "ios", "token": "djt_…", "api_url": "https://diffinery.com"}

3. A claimed job belongs to that runner; no other runner can take it. The runner starts the agent with DIFFINERY_API_URL, DIFFINERY_JOB_ID and DIFFINERY_JOB_TOKEN, and the agent fetches the job, runs it and reports back.

4. A runner runs one job at a time. To run jobs in parallel on one machine, start several runners with different --name values (each gets its own work directory).

A job that no runner claims fails after 6 hours. While a job waits, the run page shows it as queued.

If the runner can't reach Diffinery it prints warning: ... (will retry) and keeps polling. A 401, 403 or 404 from the claim request stops the runner, because retrying won't help. For a removed runner it says runner <name> was removed from <owner>/<repo>; delete <file> and register it again.

Only registered runners can claim jobs. A personal token, even a repository admin's, gets 403 with personal tokens can't claim jobs; register this runner: .... Runners used to claim that way before registration. If an older runner stops with that message, upgrade the CLI and start it again while signed in, which registers it, or register it with --registration-token.

#Manage runners

A repository can have at most 100 registered runners, each with at most 20 labels. An ephemeral runner is removed when its job finishes. Registering a runner, creating a registration token and removing a runner are recorded in the audit log.

#What a job sees on a self-hosted runner

• The job runs directly on the machine, as the user that started diffinery runner, with that user's files, network and permissions.
• The job's environment is the runner's environment, minus DIFFINERY_TOKEN, DIFFINERY_URL, DIFFINERY_RUNNER_AGENT, GH_TOKEN and GITHUB_TOKEN, plus the usual GITHUB_* and RUNNER_* variables. DIFFINERY_GITHUB_TOKEN (see below) is used by the agent to download actions but is removed from every step's environment.
• The workspace is <work dir>/<repository name> and is not wiped between jobs. actions/checkout cleans it by default (clean: true).
• The tool cache (RUNNER_TOOL_CACHE) is <work dir>/_tool unless the runner's environment sets RUNNER_TOOL_CACHE. Tools that setup-node, setup-python and setup-go download stay there for later jobs.
• runner.os reports Linux and RUNNER_ENVIRONMENT reports diffinery-hosted on every machine, including Macs. Use your own labels, or check uname in a step, if a workflow needs to know the platform.
• Remote actions (uses: owner/repo@ref) are downloaded from codeload.github.com. Set DIFFINERY_GITHUB_TOKEN in the runner's environment to authenticate those downloads (useful against GitHub's rate limits), or DIFFINERY_ACTIONS_DOWNLOAD_URL to a URL template containing {owner}, {repo} and {ref} to download from a mirror instead. If the download from codeload.github.com fails, the agent tries a shallow git fetch of the ref from https://github.com/<owner>/<repo> before failing the step with Failed to download action '<owner>/<repo>@<ref>': .... There is no git fallback when DIFFINERY_ACTIONS_DOWNLOAD_URL is set.
• The common official actions (actions/checkout, actions/cache, actions/upload-artifact, actions/download-artifact, actions/setup-node, actions/setup-python, actions/setup-go) run as built-in implementations when the version and inputs allow it. Set DIFFINERY_DISABLE_BUILTINS=1 in the runner's environment to always download and run the real actions on that runner. See Compatibility with GitHub Actions.
• setup-node, setup-python and setup-go download their tools from nodejs.org/dist, the actions/python-versions manifest and go.dev. Override the sources with DIFFINERY_NODE_MIRROR, DIFFINERY_PYTHON_MANIFEST and DIFFINERY_GO_INDEX in the runner's environment if the machine can't reach them.

#Run as a service with systemd

On Linux, run the runner as a systemd service under a dedicated user so it starts at boot and restarts after failures.

1. Create a user and put the CLI and agent where it can read them:

   sudo useradd --system --create-home --shell /bin/bash diffinery-runner
   sudo install -m 0755 ~/.local/bin/diffinery /usr/local/bin/diffinery
   sudo install -D -m 0644 containers/runner/dist/agent.mjs /opt/diffinery/agent.mjs

2. Register the runner once as that user, with a registration token from a repository admin. Stop it with Ctrl-C once it says it's waiting:

   sudo -u diffinery-runner -H env DIFFINERY_URL=https://diffinery.com DIFFINERY_RUNNER_AGENT=/opt/diffinery/agent.mjs \
     diffinery runner -R acme/website --registration-token drg_… --labels linux,gpu --name gpu-box-1

   Its credentials are saved in that user's ~/.config/diffinery/runners, so the service needs no token.

3. Create /etc/systemd/system/diffinery-runner.service:

   [Unit]
   Description=Diffinery self-hosted runner
   After=network-online.target
   Wants=network-online.target
   
   [Service]
   User=diffinery-runner
   WorkingDirectory=/home/diffinery-runner
   Environment=DIFFINERY_URL=https://diffinery.com
   Environment=DIFFINERY_RUNNER_AGENT=/opt/diffinery/agent.mjs
   ExecStart=/usr/bin/node /usr/local/bin/diffinery runner -R acme/website --labels linux,gpu --name gpu-box-1 --quiet
   Restart=always
   RestartSec=10
   
   [Install]
   WantedBy=multi-user.target

   Adjust the path to node (which node) and the repositories and labels. Pass -R explicitly: a service has no git checkout to infer the repository from.

4. Start it and follow its output:

   sudo systemctl daemon-reload
   sudo systemctl enable --now diffinery-runner
   journalctl -u diffinery-runner -f

Stopping the service sends the stop signal to the runner and the agent, which cancels the running job and reports it as cancelled. On macOS, use a launchd agent with the same command line and environment variables.

For a clean machine per job, run diffinery runner --ephemeral --registration-token drg_… inside a fresh virtual machine or container and throw it away afterwards. It exits with the job's exit code, and its token can't be used again.

#Docker on self-hosted runners

Job containers, service containers and Docker container actions work on a self-hosted runner that has Docker installed and running, with the docker CLI on the runner's PATH and permission to use the daemon (for example, membership in the docker group). Without Docker, those jobs fail with:

Job containers (jobs.<id>.container) need Docker, and this runner has no docker CLI. Use runs-on: [self-hosted, ...] on a machine with Docker installed.

or, when the CLI exists but the daemon isn't running:

Service containers (jobs.<id>.services) need Docker, and the Docker daemon on this runner is not reachable: ...

The daemon must be local to the machine, because the job container mounts the runner's work directory at the same path. Self-hosted runners don't have the registry relay that hosted runners have, so pushing an image layer larger than 100 MB to the Diffinery registry fails from a self-hosted runner. See Docker in workflows and Container registry.

#macOS, Windows and GPU machines

• macOS. The runner and agent run on macOS with Node.js 22 or later. Steps run with /bin/bash. Because the built-in setup-node, setup-python and setup-go download Linux builds, install toolchains on the Mac yourself (Homebrew, Xcode) and either skip those actions or ask for a version that's already installed (the installed one is used when it satisfies the request). Docker-based features are designed for Linux hosts.
• Windows. The runner agent runs steps with bash or sh and rejects shell: cmd with "The cmd shell is not supported on Linux runners." It reports runner.os as Linux. Native Windows isn't a supported host in this version; the practical route is to run the runner inside WSL 2 and call Windows programs from steps through WSL interop (for example powershell.exe).
• GPUs. Start the runner with a label such as gpu and run steps directly on the machine. To use the GPU from a job container, install the NVIDIA Container Toolkit on the machine and pass the Docker option through: container: { image: nvidia/cuda:12.4.1-base-ubuntu22.04, options: --gpus all }.

#Security considerations

A self-hosted runner runs code from your repository's workflows on your machine. Before you start one:

• Anyone who can push can run code on the machine. People and agents with write access can push a branch or open a pull request whose workflow targets self-hosted, and push and pull_request runs use the workflow file from that commit. Only serve repositories whose writers you trust with the machine.
• Jobs run as the runner's user. A job can read anything that user can, including the runner's saved registration and other repositories' checkouts in the work directory. A runner registered with a registration token keeps only its runner token, which can claim that repository's jobs and nothing else: it can't read the repository, push, or call the API. Someone who steals it can still take that runner's future jobs, with their secrets, until you remove the runner. So use a dedicated OS user with no other access, don't sign in with a personal token on the runner machine, and prefer --ephemeral in a throwaway VM for anything sensitive.
• Jobs get the repository's secrets, just like on hosted runners, and can reach whatever the machine can reach on your network. A job with environment: also gets that environment's secrets when its branch policy and the rules for environment secrets allow it, so a runner that serves deploy jobs handles production credentials. See Deployments and environments.
• State persists. The work directory, tool cache and anything a job writes elsewhere stay on the machine. A malicious job can leave something behind for the next job. For untrusted workloads, use --ephemeral in a throwaway VM: its token ends when it takes its one job.
• Cancelling isn't instant on your machine. When a run is cancelled, Diffinery marks the job cancelled and revokes its job token right away, but the agent finishes executing the current steps locally (their logs and results are discarded). Stop the runner (Ctrl-C) to stop the work immediately; a second Ctrl-C kills the agent.

#Troubleshooting