codician/diffinerypublic

---

#title: Runners description: Where Diffinery workflow jobs run: hosted Linux runners in Cloudflare Containers, and self-hosted runners you start with diffinery runner.

A runner is the machine that executes one workflow job. Diffinery offers two kinds: hosted runners, which run each job in a fresh Ubuntu 24.04 container on Cloudflare, and self-hosted runners, which run jobs on a machine you control (a Mac, a GPU box, a server on a private network) using the diffinery runner command. You pick one per job with runs-on, the same key GitHub Actions uses.

#How runs-on picks a runner

Diffinery reads the job's runs-on labels (case-insensitive) and decides:

1. If any label is self-hosted, the job waits for a self-hosted runner that offers every label in runs-on. See Self-hosted runners.
2. Otherwise, if every label is one Diffinery hosts, the job runs on a hosted runner.
3. Otherwise the job fails at once with:

No Diffinery runner matches runs-on: windows-latest. Linux (ubuntu-*) runners are hosted; anything else needs runs-on: [self-hosted, ...] and a registered runner.

Things to know:

• Every ubuntu-* label gets the same Ubuntu 24.04 x64 image, including ubuntu-22.04 and labels that ask for Arm, such as ubuntu-24.04-arm.
• runs-on can be a string, a list, or a mapping with labels. A group in the mapping form is ignored; only labels count.
• runs-on is evaluated for each matrix combination, so an entry can be an expression using github, inputs, vars, needs, matrix and strategy. An expression can give one label or a list of labels. Empty and null values are dropped.
• If evaluating runs-on fails, only that matrix combination fails, with Error evaluating the job for this matrix combination: <message>. If no labels are left, the job fails with No Diffinery runner matches runs-on: (no labels). ....

jobs:
  test:
    runs-on: ubuntu-latest        # hosted
    steps:
      - run: echo "hosted"
  build:
    strategy:
      matrix:
        target: [linux, macos]
    # The linux leg runs on a hosted runner; the macos leg waits for a
    # self-hosted runner with the macos label.
    runs-on: ${{ matrix.target == 'macos' && fromJSON('["self-hosted","macos"]') || 'ubuntu-latest' }}
    steps:
      - run: uname -a
  gpu:
    runs-on: [self-hosted, linux, gpu]   # waits for a runner started with --labels linux,gpu
    steps:
      - run: nvidia-smi

#Hosted runners

Each job on a hosted runner gets its own container on Cloudflare Containers, started for that job and thrown away when it finishes. Nothing carries over between jobs except what you save with caches and artifacts (see Caching and artifacts).

#Machine

#The Ubuntu 24.04 image

The hosted image is built from containers/runner/Dockerfile in the Diffinery source. It is deliberately small. This is the complete list of what is preinstalled:

Image settings that affect your steps:

• git config --system safe.directory '*' is set, and Git LFS is installed system-wide.
• PIP_BREAK_SYSTEM_PACKAGES=1 is set, so pip install works without a virtual environment, as on GitHub's hosted runners.
• COREPACK_ENABLE_DOWNLOAD_PROMPT=0 is set, so corepack never stops to ask.
• ImageOS=ubuntu24 and ImageVersion=20250101.1 are set, as on GitHub's ubuntu-24.04 image. actions/setup-go and actions/setup-python include them in their cache keys.
• zstd is installed, so actions/cache compresses with zstd, as on GitHub.
• The apt package lists are removed to keep the image small. Run apt-get update before apt-get install.

Anything else (Go, Java, Ruby, Rust, .NET, PowerShell, browsers, cloud CLIs) is not preinstalled. Install it in a step:

steps:
  - uses: actions/setup-go@v5          # downloads Go from go.dev
    with:
      go-version: "1.23"
  - run: apt-get update && apt-get install -y --no-install-recommends openjdk-17-jdk-headless
  - run: pip install awscli

actions/setup-node, actions/setup-python and actions/setup-go use the preinstalled Node.js or Python when its version satisfies the one you ask for, and download the version otherwise. See Compatibility with GitHub Actions for their supported inputs.

Note: Because jobs run as root and sudo isn't installed, sudo apt-get install ... fails with sudo: command not found. Drop the sudo.

#Docker on hosted runners

Docker works on hosted runners the same way as on GitHub's: job containers, service containers, Docker container actions, and docker build, docker run and docker compose in run: steps. Details:

• Started on first use. The docker command is a wrapper that starts the Docker daemon the first time anything in the job needs it (a docker command in a step, a job container:, services:, or a Docker action). The first use prints Starting the Docker daemon and takes a few seconds; jobs that never touch Docker don't pay for it. If the daemon can't start, the step fails with The Docker daemon did not start: followed by the end of the daemon's log.
• Docker Hub through mirror.gcr.io. The daemon uses https://mirror.gcr.io as its registry mirror, so pulls of Docker Hub images (redis:7-alpine, postgres:16) usually come from Google's public mirror and avoid Docker Hub's anonymous rate limits. Images the mirror doesn't have are pulled from Docker Hub as usual.
• DNS. The daemon gives containers the DNS servers 1.1.1.1 and 8.8.8.8.
• Pushing to the Diffinery registry. The daemon and BuildKit builders reach diffinery.com through a local relay that splits image layer uploads into chunks, so layers larger than Cloudflare's 100 MB request limit push from hosted workflows. See Container registry.
• Buildx builders. docker buildx create with the docker-container driver (which docker/setup-buildx-action runs) is handled by the wrapper: it starts the BuildKit container on the host network and registers it as a remote builder, because the normal driver can't work inside a Cloudflare Container. docker buildx rm removes it.

The full guide is Docker in workflows.

#Environment variables

Hosted and self-hosted runners set the GitHub-compatible variables (CI=true, GITHUB_ACTIONS=true, GITHUB_REPOSITORY, GITHUB_SHA, GITHUB_REF, GITHUB_WORKSPACE, GITHUB_ENV, GITHUB_OUTPUT, and so on) plus DIFFINERY=true. GITHUB_SERVER_URL is https://diffinery.com and GITHUB_API_URL is https://diffinery.com/api/v3. RUNNER_NAME is diffinery and RUNNER_ENVIRONMENT is diffinery-hosted on every runner. Workflows can override CI and DIFFINERY, but not GITHUB_* or RUNNER_* variables.

Other variables a job gets:

• ImageOS and ImageVersion, on hosted and self-hosted runners alike (ubuntu24 and 20250101.1 unless the runner's environment sets them).
• RUNNER_DEBUG=1 when the secret or variable ACTIONS_STEP_DEBUG or ACTIONS_RUNNER_DEBUG is true. See Expressions and contexts.
• ACTIONS_RUNTIME_TOKEN, ACTIONS_RUNTIME_URL, ACTIONS_RESULTS_URL, ACTIONS_CACHE_URL and ACTIONS_CACHE_SERVICE_V2, but only in JavaScript actions, Docker container actions and the built-in actions, never in run: steps. They let actions such as actions/cache and actions/upload-artifact use the cache and artifact services. See Caching and artifacts.

#Limits and timeouts

If the container stops unexpectedly (for example it runs out of memory), the job fails with The runner stopped unexpectedly (exit code N).

#Windows and macOS

Cloudflare Containers run Linux only, so Diffinery has no Windows or macOS hosted runners. runs-on: windows-latest or runs-on: macos-latest fails with the "No Diffinery runner matches runs-on" message. For macOS builds, Windows-specific tooling or GPUs, use a self-hosted runner on your own machine with runs-on: [self-hosted, ...].

#Self-hosted runners

A self-hosted runner is the diffinery runner command running on a machine you control. It polls one or more repositories for waiting jobs whose runs-on includes self-hosted, claims one, runs it with the Diffinery runner agent (the same program that runs inside hosted containers), streams the logs to Diffinery, and reports the result. Use it for:

• macOS: build and sign iOS or macOS apps with Xcode on a Mac.
• GPUs: run CUDA tests or model evaluation on a machine with a GPU.
• Private networks: deploy to or test against servers that Cloudflare can't reach (a database behind a VPN, an on-premises cluster).
• Long-lived caches and big toolchains: the runner's work directory and tool cache persist between jobs.

#Requirements

• Node.js 22 or later on the machine.

• The diffinery CLI. Install it with:

  curl -fsSL https://diffinery.com/-/cli/diffinery.mjs -o ~/.local/bin/diffinery && chmod +x ~/.local/bin/diffinery

• The runner agent, a single Node.js file named agent.mjs. Build it from a checkout of the Diffinery source:

  pnpm install
  pnpm --filter @diffinery/runner build      # produces containers/runner/dist/agent.mjs

  Copy containers/runner/dist/agent.mjs to the runner machine (for example to /opt/diffinery/agent.mjs) and pass it with --agent or the DIFFINERY_RUNNER_AGENT environment variable.

• A token with the admin scope, belonging to a person who has the admin role on each repository the runner serves. Claiming jobs requires admin access; a weaker role or token scope gets 403 admin access required. Agents can't run self-hosted runners, because an agent's role is capped at write. Create the token under Settings → Access tokens, and restrict it to the repositories the runner serves (see Personal access tokens).

• Bash (the default shell for run: steps; sh is used when bash is missing), git and tar, which the built-in actions use.

• Docker, only if jobs use container:, services: or Docker actions.

#Start a runner

1. Log in once, or set DIFFINERY_TOKEN (and DIFFINERY_URL if you don't use https://diffinery.com):

   diffinery login --token dfy_...

2. Start the runner for one or more repositories:

   diffinery runner -R acme/website --labels macos,arm64 --agent /opt/diffinery/agent.mjs

   It prints the labels it offers and waits:

   Runner my-mac offering [self-hosted, macos, arm64] for acme/website
   Waiting for jobs. Press Ctrl-C to stop.

3. Target it from a workflow:

   jobs:
     ios:
       runs-on: [self-hosted, macos, arm64]
       steps:
         - uses: actions/checkout@v4
         - run: xcodebuild -scheme App -destination 'platform=iOS Simulator,name=iPhone 16' test

When a job arrives the runner prints → acme/website: running job ios (run <run id>, job <job id>), echoes the step output prefixed with the job name, and ends with ✓ job ios finished or X job ios finished with exit code 1.

#diffinery runner options

The full generated reference is at CLI reference.

#Labels and job matching

• A job is eligible when its runs-on contains self-hosted and every one of its labels is offered by the runner. The runner offers self-hosted plus everything in --labels. Comparison is case-insensitive.
• A runner with more labels than a job asks for can still take the job: --labels linux,gpu,cuda12 takes runs-on: [self-hosted, gpu].
• runs-on: [self-hosted] alone can be taken by any runner serving the repository.
• When several jobs are waiting, a runner takes the oldest run's jobs first.
• Labels have no meaning to Diffinery beyond matching. Choose names that describe the machine (macos, arm64, gpu, vpn-prod).

#How jobs are claimed

1. When a self-hosted job is ready to start, Diffinery marks it waiting and creates a queued check on the commit.

2. Each runner polls POST /api/v3/repos/{owner}/{repo}/actions/runners/claim with its name and labels, every --interval seconds per repository while idle. The response is 204 No Content when nothing matches, or the job with a fresh job token:

   POST /api/v3/repos/acme/website/actions/runners/claim
   Authorization: Bearer dfy_...
   Content-Type: application/json
   
   {"name": "my-mac", "labels": ["macos", "arm64"]}

   {"job_id": "…", "run_id": "…", "name": "ios", "token": "djt_…", "api_url": "https://diffinery.com"}

3. A claimed job belongs to that runner; no other runner can take it. The runner starts the agent with DIFFINERY_API_URL, DIFFINERY_JOB_ID and DIFFINERY_JOB_TOKEN, and the agent fetches the job, runs it and reports back.

4. A runner runs one job at a time. To run jobs in parallel on one machine, start several runners with different --name values (each gets its own work directory).

A job that no runner claims fails after 6 hours. While a job waits, the run page shows it as queued.

If the runner can't reach Diffinery it prints warning: ... (will retry) and keeps polling. A 401, 403 or 404 from the claim request stops the runner, because retrying won't help (wrong token, missing admin access, or a mistyped repository).

#What a job sees on a self-hosted runner

• The job runs directly on the machine, as the user that started diffinery runner, with that user's files, network and permissions.
• The job's environment is the runner's environment, minus DIFFINERY_TOKEN, DIFFINERY_URL, DIFFINERY_RUNNER_AGENT, GH_TOKEN and GITHUB_TOKEN, plus the usual GITHUB_* and RUNNER_* variables. DIFFINERY_GITHUB_TOKEN (see below) is used by the agent to download actions but is removed from every step's environment.
• The workspace is <work dir>/<repository name> and is not wiped between jobs. actions/checkout cleans it by default (clean: true).
• The tool cache (RUNNER_TOOL_CACHE) is <work dir>/_tool unless the runner's environment sets RUNNER_TOOL_CACHE. Tools that setup-node, setup-python and setup-go download stay there for later jobs.
• runner.os reports Linux and RUNNER_ENVIRONMENT reports diffinery-hosted on every machine, including Macs. Use your own labels, or check uname in a step, if a workflow needs to know the platform.
• Remote actions (uses: owner/repo@ref) are downloaded from codeload.github.com. Set DIFFINERY_GITHUB_TOKEN in the runner's environment to authenticate those downloads (useful against GitHub's rate limits), or DIFFINERY_ACTIONS_DOWNLOAD_URL to a URL template containing {owner}, {repo} and {ref} to download from a mirror instead. If the download from codeload.github.com fails, the agent tries a shallow git fetch of the ref from https://github.com/<owner>/<repo> before failing the step with Failed to download action '<owner>/<repo>@<ref>': .... There is no git fallback when DIFFINERY_ACTIONS_DOWNLOAD_URL is set.
• The common official actions (actions/checkout, actions/cache, actions/upload-artifact, actions/download-artifact, actions/setup-node, actions/setup-python, actions/setup-go) run as built-in implementations when the version and inputs allow it. Set DIFFINERY_DISABLE_BUILTINS=1 in the runner's environment to always download and run the real actions on that runner. See Compatibility with GitHub Actions.
• setup-node, setup-python and setup-go download their tools from nodejs.org/dist, the actions/python-versions manifest and go.dev. Override the sources with DIFFINERY_NODE_MIRROR, DIFFINERY_PYTHON_MANIFEST and DIFFINERY_GO_INDEX in the runner's environment if the machine can't reach them.

#Run as a service with systemd

On Linux, run the runner as a systemd service under a dedicated user so it starts at boot and restarts after failures.

1. Create a user and put the CLI and agent where it can read them:

   sudo useradd --system --create-home --shell /bin/bash diffinery-runner
   sudo install -m 0755 ~/.local/bin/diffinery /usr/local/bin/diffinery
   sudo install -D -m 0644 containers/runner/dist/agent.mjs /opt/diffinery/agent.mjs

2. Store the token in a file only root can read:

   echo 'DIFFINERY_TOKEN=dfy_...' | sudo tee /etc/diffinery-runner.env >/dev/null
   sudo chmod 600 /etc/diffinery-runner.env

3. Create /etc/systemd/system/diffinery-runner.service:

   [Unit]
   Description=Diffinery self-hosted runner
   After=network-online.target
   Wants=network-online.target
   
   [Service]
   User=diffinery-runner
   WorkingDirectory=/home/diffinery-runner
   EnvironmentFile=/etc/diffinery-runner.env
   Environment=DIFFINERY_URL=https://diffinery.com
   Environment=DIFFINERY_RUNNER_AGENT=/opt/diffinery/agent.mjs
   ExecStart=/usr/bin/node /usr/local/bin/diffinery runner -R acme/website --labels linux,gpu --name gpu-box-1 --quiet
   Restart=always
   RestartSec=10
   
   [Install]
   WantedBy=multi-user.target

   Adjust the path to node (which node) and the repositories and labels. Pass -R explicitly: a service has no git checkout to infer the repository from.

4. Start it and follow its output:

   sudo systemctl daemon-reload
   sudo systemctl enable --now diffinery-runner
   journalctl -u diffinery-runner -f

Stopping the service sends the stop signal to the runner and the agent, which cancels the running job and reports it as cancelled. On macOS, use a launchd agent with the same command line and environment variables.

For a clean machine per job, run diffinery runner --once inside a fresh virtual machine or container and throw it away afterwards; --once exits with the job's exit code.

#Docker on self-hosted runners

Job containers, service containers and Docker container actions work on a self-hosted runner that has Docker installed and running, with the docker CLI on the runner's PATH and permission to use the daemon (for example, membership in the docker group). Without Docker, those jobs fail with:

Job containers (jobs.<id>.container) need Docker, and this runner has no docker CLI. Use runs-on: [self-hosted, ...] on a machine with Docker installed.

or, when the CLI exists but the daemon isn't running:

Service containers (jobs.<id>.services) need Docker, and the Docker daemon on this runner is not reachable: ...

The daemon must be local to the machine, because the job container mounts the runner's work directory at the same path. Self-hosted runners don't have the registry relay that hosted runners have, so pushing an image layer larger than 100 MB to the Diffinery registry fails from a self-hosted runner. See Docker in workflows and Container registry.

#macOS, Windows and GPU machines

• macOS. The runner and agent run on macOS with Node.js 22 or later. Steps run with /bin/bash. Because the built-in setup-node, setup-python and setup-go download Linux builds, install toolchains on the Mac yourself (Homebrew, Xcode) and either skip those actions or ask for a version that's already installed (the installed one is used when it satisfies the request). Docker-based features are designed for Linux hosts.
• Windows. The runner agent runs steps with bash or sh and rejects shell: cmd with "The cmd shell is not supported on Linux runners." It reports runner.os as Linux. Native Windows isn't a supported host in this version; the practical route is to run the runner inside WSL 2 and call Windows programs from steps through WSL interop (for example powershell.exe).
• GPUs. Start the runner with a label such as gpu and run steps directly on the machine. To use the GPU from a job container, install the NVIDIA Container Toolkit on the machine and pass the Docker option through: container: { image: nvidia/cuda:12.4.1-base-ubuntu22.04, options: --gpus all }.

#Security considerations

A self-hosted runner runs code from your repository's workflows on your machine. Before you start one:

• Anyone who can push can run code on the machine. People and agents with write access can push a branch or open a pull request whose workflow targets self-hosted, and push and pull_request runs use the workflow file from that commit. Only serve repositories whose writers you trust with the machine.
• Jobs run as the runner's user. A job can read anything that user can, including the runner's own token (from the CLI's config file ~/.config/diffinery/config.json or the service's environment) and other repositories' checkouts in the work directory. Use a dedicated OS user with no other access, and a token restricted to the repositories the runner serves.
• Jobs get the repository's secrets, just like on hosted runners, and can reach whatever the machine can reach on your network. A job with environment: also gets that environment's secrets when its branch policy and the rules for environment secrets allow it, so a runner that serves deploy jobs handles production credentials. See Deployments and environments.
• State persists. The work directory, tool cache and anything a job writes elsewhere stay on the machine. A malicious job can leave something behind for the next job. For untrusted workloads, use --once in a throwaway VM.
• Cancelling isn't instant on your machine. When a run is cancelled, Diffinery marks the job cancelled and revokes its job token right away, but the agent finishes executing the current steps locally (their logs and results are discarded). Stop the runner (Ctrl-C) to stop the work immediately; a second Ctrl-C kills the agent.

#Troubleshooting