Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Jobs a rollout stops before their steps ran start again; hosted runners have passwordless sudo (#154)

* A job whose runner a rollout replaces before its steps ran starts again; later it fails and says why

The platform can still replace a hosted runner while its job runs, even with
rollout_active_grace_period set. A job on an old-image container started during
the d10062f rollout was stopped three minutes in.

- The agent cancels the rest of the job once the platform's stop signal ends a step,
  and marks its result `replaced`.
- If no step but actions/checkout had started, Diffinery puts the job back in line
  and starts it again from the beginning in a new container, up to
  MAX_RUNNER_RESTARTS (2) times. The log starts over with a line that says so.
- Once another step has started, the job fails: "The platform replaced this job's
  runner with a new version while step X ran", since running that step twice could
  repeat a deploy or publish.
- A container on a replaced image that stops without reporting is handled the same
  way. A late stop of a replaced container no longer touches the restarted job.
- The jobs API returns `restarts`, and the Live test prints it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ruyx3Qr4jYrZEpPi466deb

* Hosted runners have passwordless sudo, as on GitHub

Jobs run as root, but the image had no sudo, so the common `sudo apt-get install`
failed with "sudo: command not found". The image now installs sudo (root needs no
password). The Live test's hosted jobs run `sudo -n true` and check that it printed
"sudo ok 0". The docs no longer tell people to drop sudo.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ruyx3Qr4jYrZEpPi466deb

* The jobs API reference names restarts

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ruyx3Qr4jYrZEpPi466deb

---------

Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent a6ebb4ccommit f60a1c39395035693415ea03139c9433914b331bBrowse files
16 files changed+223−37
Files (16)
modified.github/workflows/live-test.yml+5−1View file
@@ -245,6 +245,8 @@
245245 steps:
246246 - uses: actions/checkout@v4
247247 - run: echo "live ok ${{ '${{' }} matrix.n }} $(cat README.md)"
248+ # Passwordless sudo, as on GitHub's runners (`sudo apt-get install ...` is common).
249+ - run: sudo -n true && echo "sudo ok $(sudo -n id -u)"
248250 docker:
249251 runs-on: ubuntu-latest
250252 services:
@@ -409,7 +411,8 @@
409411 id=$(jq -r .id <<<"$run")
410412 jobs=$(api "/repos/$OWNER/$REPO/actions/runs/$id/jobs")
411413 # placementTries: times a job went back in line (no machine, or a runner on a replaced image turned away).
412- jq -r '.jobs[] | " \(.name): \(.conclusion)\(if .failure then " - " + (.failure.message // "") else "" end)\(try " (\((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601))s)" catch "")\(if (.placementTries // 0) > 0 then ", started again \(.placementTries) time(s)" else "" end)"' <<<"$jobs"
414+ # restarts: times it started over because the platform replaced its runner before its steps ran.
415+ jq -r '.jobs[] | " \(.name): \(.conclusion)\(if .failure then " - " + (.failure.message // "") else "" end)\(try " (\((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601))s)" catch "")\(if (.placementTries // 0) > 0 then ", started again \(.placementTries) time(s)" else "" end)\(if (.restarts // 0) > 0 then ", restarted \(.restarts) time(s) after its runner was replaced" else "" end)"' <<<"$jobs"
413416 if [ "$(jq -r .conclusion <<<"$run")" != success ]; then
414417 for job in $(jq -r '.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out") | .id' <<<"$jobs"); do
415418 echo "::group::Last log lines of job $job"
@@ -428,6 +431,7 @@
428431 for job in $(jq -r '.jobs[] | select(.name | startswith("hosted")) | .id' <<<"$jobs"); do
429432 log=$(api "/repos/$OWNER/$REPO/actions/jobs/$job/logs")
430433 grep -F "live ok" <<<"$log" | grep -F "# $REPO" || { echo "::error::job $job did not print its line (checkout or run step failed)"; exit 1; }
434+ grep -F "sudo ok 0" <<<"$log" || { echo "::error::job $job could not run sudo without a password"; exit 1; }
431435 done
432436 job=$(jq -r '.jobs[] | select(.name == "docker") | .id' <<<"$jobs")
433437 log=$(api "/repos/$OWNER/$REPO/actions/jobs/$job/logs")
modifiedapps/diffinery/docs/actions/compatibility.md+2−3View file
@@ -88,7 +88,7 @@
8888 What's different about the hosted image:
8989
9090 - It's much smaller than GitHub's image. Git, Git LFS, curl, jq, zstd, build-essential, Python 3.12, Node.js 22 with npm, pnpm and yarn, nvm, Docker and the GitHub CLI (`gh`) are preinstalled. `ImageOS` is `ubuntu24`, as on GitHub's `ubuntu-24.04` image. Go, Java, Ruby, Rust, .NET, PowerShell, browsers and cloud CLIs aren't. See [The Ubuntu 24.04 image](/docs/actions/runners#the-ubuntu-2404-image).
91-- Jobs run as `root` and `sudo` isn't installed. Drop `sudo` from commands.
91+- Jobs run as `root`. `sudo` is installed and needs no password, as on GitHub, so `sudo` in commands keeps working.
9292 - The apt package lists are removed. Run `apt-get update` before `apt-get install`.
9393 - Every job starts in a fresh container. Nothing carries over between jobs except caches and artifacts.
9494
@@ -382,7 +382,7 @@
382382 1. Run `diffinery workflow validate` on each file and fix any errors. Read the warnings for ignored keys.
383383 2. Push and watch the first run with `diffinery run watch` or on the **Actions** tab.
384384 3. Replace `runs-on: windows-*` and `macos-*` with Linux, or with `[self-hosted, ...]` labels and a [self-hosted runner](/docs/actions/runners#self-hosted-runners). This includes `matrix.os` values used in `runs-on`.
385-4. Remove `sudo`, and add `apt-get update` before `apt-get install`.
385+4. Add `apt-get update` before `apt-get install`.
386386 5. Install tools the hosted image doesn't have (Go, Java, Ruby, Rust, PowerShell, cloud CLIs) with a setup action or `apt-get`.
387387 6. Check what each job does with `GITHUB_TOKEN`. In a new repository the default token only reads, so give jobs that push, tag, comment, label or merge the `permissions:` they need, or have an admin set the repository default to write (see [GITHUB_TOKEN permissions](/docs/actions/workflow-permissions)). Jobs that change workflow files, or that open or approve pull requests without the repository allowing it, need a personal or agent token from a secret.
388388 7. Give `actions/checkout` of another private repository a token from a secret.
@@ -408,7 +408,6 @@
408408 | `Can't find 'action.yml' or 'action.yaml' under '<dir>'. ...` | Check out the repository before using a local action, and check the path. |
409409 | `Unsupported action runtime 'runs.using: <value>' in <action>` | The action uses a runtime other than `node*`, `composite` or `docker`. |
410410 | ``The `cmd` shell is not supported on Linux runners.`` | Use `bash` or `sh`. |
411-| `sudo: command not found` | Jobs run as root on hosted runners. Remove `sudo`. |
412411 | `Failed to fetch ref '<ref>' from <owner>/<repo>. Check that the ref exists and the token has read access.` | Checking out another private repository needs a token from a secret, the job lacks `contents: read`, or the ref doesn't exist. |
413412 | `No go-version specified and Go is not installed on this runner.` | Set `go-version` or `go-version-file`. |
414413 | `jobs.<id>: a matrix on a job that calls a reusable workflow is not supported yet` | Move the matrix into the called workflow, or write one calling job per value. |
modifiedapps/diffinery/docs/actions/runners.md+5−6View file
@@ -65,7 +65,7 @@
6565 | Platform | Cloudflare Containers, instance type `standard-2` (see Cloudflare's Containers documentation for its vCPU, memory and disk) |
6666 | Architecture | `linux/amd64` (`runner.arch` is `X64`) |
6767 | Operating system | Ubuntu 24.04 (`runner.os` is `Linux`) |
68-| User | `root` (there is no `sudo` command; you don't need it) |
68+| User | `root`. `sudo` is installed and needs no password, so `sudo apt-get install ...` works as on GitHub |
6969 | Network | Outbound internet access (only HTTP and HTTPS on instances that set [`EGRESS=web`](/docs/reference/limits#egress)); no inbound connections |
7070 | Workspace | `/workspace/<repository name>` (`GITHUB_WORKSPACE`) |
7171 | Temp directory | `/workspace/_temp` (`RUNNER_TEMP`) |
@@ -79,7 +79,7 @@
7979
8080 | Group | Installed |
8181 |---|---|
82-| Base tools | `ca-certificates`, `curl`, `git`, `git-lfs`, `jq`, `zip`, `unzip`, `xz-utils`, `tar`, `gzip`, `zstd`, `pkg-config`, `openssh-client`, `gnupg` |
82+| Base tools | `ca-certificates`, `curl`, `git`, `git-lfs`, `jq`, `zip`, `unzip`, `xz-utils`, `tar`, `gzip`, `zstd`, `sudo`, `pkg-config`, `openssh-client`, `gnupg` |
8383 | GitHub CLI | `gh`, from GitHub's apt repository |
8484 | Build tools | `build-essential` (gcc, g++, make and the C library headers) |
8585 | Libraries for Ruby and native extensions | `libyaml-dev`, `libssl-dev`, `libffi-dev`, `zlib1g-dev`, `libreadline-dev`, `libgmp-dev`, so `ruby/setup-ruby`'s prebuilt Rubies run and gems with C extensions build |
@@ -109,7 +109,7 @@
109109
110110 `actions/setup-node`, `actions/setup-python` and `actions/setup-go` use the preinstalled Node.js or Python when its version satisfies the one you ask for, and download the version otherwise. See [Compatibility with GitHub Actions](/docs/actions/compatibility) for their supported inputs.
111111
112-> **Note:** Because jobs run as `root` and `sudo` isn't installed, `sudo apt-get install ...` fails with `sudo: command not found`. Drop the `sudo`.
112+> **Note:** Jobs run as `root`, and `sudo` runs commands without asking for a password, as on GitHub's runners. The image ships without apt package lists, so run `sudo apt-get update` before `sudo apt-get install ...`.
113113
114114 ### Docker on hosted runners
115115
@@ -143,11 +143,11 @@
143143 | Matrix size | 256 jobs per matrix | The job fails with "A matrix may produce at most 256 jobs." |
144144 | Log line length | 16,384 characters | Longer lines are cut and end with `…` |
145145 | Log size | 16 MiB of text per job | The log ends with "The log is cut off here: a job's log is limited to 16 MiB of text." Later lines are dropped; the job keeps running. |
146-| Platform updates | A deploy doesn't replace the container of a running job: running jobs finish on the version they started with. If the platform replaces a container anyway, its stop signal reaches the running step too, so that step fails and the job needs a re-run | The log shows "This runner is being replaced by a new version" and, on the step, "The platform stopped this step's processes while replacing the runner with a new version." |
146+| Platform updates | A deploy is set up not to replace the container of a running job, so running jobs normally finish on the version they started with. The platform doesn't guarantee it, though. If it replaces a container anyway, its stop signal reaches the running step too. If no step but `actions/checkout` had started, Diffinery starts the job again from the beginning on a new runner, up to twice. Once another step has started, the job fails instead, because running that step twice could repeat what it did (a deploy, a publish). Re-run it from the run page | The log shows "This runner is being replaced by a new version" and, on the step, "The platform stopped this step's processes while replacing the runner with a new version." A restarted job's log starts over with "Diffinery started the job again from the beginning on a new runner (restart N of 2)". A failed one says "The platform replaced this job's runner with a new version while step … ran" |
147147
148148 If the container stops unexpectedly (for example it runs out of memory), the job fails with `The runner stopped unexpectedly (exit code N).` A container that stops before its runner fetched the job is replaced instead, since no step ran.
149149
150-The log's first lines name where the job ran, for example `Runner: image diffinery-runner:96dbbe31790f, in fra06, EEUR, container 6f1c…`. The jobs API returns the same as `runner` (`image`, `location`, `instance`), next to `runner_name`. `GET /meta` names the image the last deploy set as `runner_image`. While a deploy's rollout is updating instances, a new container can still start on the previous image. Diffinery turns that runner away before it runs anything and starts the job again on another container (after 30 s, then 1, 2 and 4 minutes). After four tries, the job runs on whatever image it got rather than wait for a rollout that has stalled. The jobs API's `placementTries` counts the times a job went back in line before a runner took it.
150+The log's first lines name where the job ran, for example `Runner: image diffinery-runner:96dbbe31790f, in fra06, EEUR, container 6f1c…`. The jobs API returns the same as `runner` (`image`, `location`, `instance`), next to `runner_name`. `GET /meta` names the image the last deploy set as `runner_image`. While a deploy's rollout is updating instances, a new container can still start on the previous image. Diffinery turns that runner away before it runs anything and starts the job again on another container (after 30 s, then 1, 2 and 4 minutes). After four tries, the job runs on whatever image it got rather than wait for a rollout that has stalled. The jobs API's `placementTries` counts the times a job went back in line before a runner took it, and `restarts` counts the times it started again because the platform replaced its runner.
151151
152152 ### Windows and macOS
153153
@@ -396,7 +396,6 @@
396396 | A self-hosted job stays queued | No runner serving that repository offers every label. Compare the job's `runs-on` with the labels the runner printed at start. The job fails after 6 hours. |
397397 | `403 admin access required` | The runner's token or account lacks admin on the repository, or the token isn't `admin`-scoped. |
398398 | `no runner agent given` or `runner agent not found at ...` | Pass `--agent /path/to/agent.mjs` or set `DIFFINERY_RUNNER_AGENT`. Build it with `pnpm --filter @diffinery/runner build`. |
399-| `sudo: command not found` on a hosted runner | Jobs run as root; remove `sudo`. |
400399 | `E: Unable to locate package ...` | Run `apt-get update` first; the image ships without package lists. |
401400 | `The runner stopped unexpectedly (exit code N).` | The hosted container exited, often from running out of memory. Reduce parallelism in the job (for example `make -j2`) or move it to a self-hosted runner. |
402401 | `Connection refused` or `Connection timed out` on SSH (`git@...`), SMTP or a database port, on a hosted runner | The instance sets [`EGRESS=web`](/docs/self-hosting/configuration#egress), which lets hosted jobs use only HTTP and HTTPS. Use an `https://` remote or API, or run the job on a self-hosted runner. diffinery.com doesn't set it. |
modifiedapps/diffinery/src/docs/api-meta.ts+2−2View file
@@ -1295,7 +1295,7 @@
12951295 description: "A zip with one text file per job. Jobs past 24 MB of logs in total are left out.",
12961296 access: "read",
12971297 },
1298- "GET /repos/{owner}/{repo}/actions/runs/{id}/jobs": { summary: "List a run's jobs", description: "Each job as `GET .../actions/jobs/{id}` returns it, with `runner` and `placementTries`.", access: "read" },
1298+ "GET /repos/{owner}/{repo}/actions/runs/{id}/jobs": { summary: "List a run's jobs", description: "Each job as `GET .../actions/jobs/{id}` returns it, with `runner`, `placementTries` and `restarts`.", access: "read" },
12991299 "GET /repos/{owner}/{repo}/actions/runs/{id}/pending_deployments": {
13001300 summary: "List the environments a run waits on",
13011301 description: "Environments whose required reviewers or wait timer hold the run's jobs, with `current_user_can_approve` for the caller. See [Required reviewers and wait timers](/docs/actions/environments#required-reviewers-and-wait-timers).",
@@ -1329,7 +1329,7 @@
13291329 status: 201,
13301330 body: { enable_debug_logging: ["boolean", "Accepted and ignored."] },
13311331 },
1332- "GET /repos/{owner}/{repo}/actions/jobs/{id}": { summary: "Get a job", description: "The job with its steps. GitHub's job fields, plus Diffinery's own: `runner` (the hosted container's `image`, `location` and `instance`; `null` on a self-hosted runner) and `placementTries` (how many times a hosted job went back in line before a runner took it: no free container, or a runner on a replaced image). See /docs/actions/runners.", access: "read" },
1332+ "GET /repos/{owner}/{repo}/actions/jobs/{id}": { summary: "Get a job", description: "The job with its steps. GitHub's job fields, plus Diffinery's own: `runner` (the hosted container's `image`, `location` and `instance`; `null` on a self-hosted runner) and `placementTries` (how many times a hosted job went back in line before a runner took it: no free container, or a runner on a replaced image) and `restarts` (how many times it started over because the platform replaced its runner before its steps ran). See /docs/actions/runners.", access: "read" },
13331333 "POST /repos/{owner}/{repo}/actions/jobs/{id}/rerun": {
13341334 summary: "Re-run a job",
13351335 description:
modifiedapps/diffinery/src/repo/actions.ts+72−4View file
@@ -67,6 +67,13 @@
6767 */
6868 export const STALE_IMAGE_TRIES = 4;
6969 /**
70+ * A deploy's container rollout can stop a hosted runner while its job runs. A job stopped before
71+ * any of its steps but checkout ran is started again from the beginning on a new runner, this many
72+ * times. One stopped later fails instead and says so: its steps may have had effects (a deploy, a
73+ * publish) that running them again would repeat.
74+ */
75+export const MAX_RUNNER_RESTARTS = 2;
76+/**
7077 * A container start that failed on the platform's side and works when tried again: no machine free,
7178 * a container sidecar or Durable Object restarting under it (seen during rollouts as "Container
7279 * sidecar is shutting down"), or a dropped connection to it.
@@ -187,6 +194,7 @@
187194 runner_info: string | null;
188195 /** Set on a job a partial re-run copied from an earlier attempt: the job that actually ran. */
189196 carried_from?: string | null;
197+ restarts: number;
190198 };
191199
192200 /** Where a job ran, as its runner reported it: the runner image, the container's location and instance. */
@@ -262,6 +270,8 @@
262270 carriedFrom: string | null;
263271 /** Times a hosted job went back in line before a runner took it (no machine, or a runner on a replaced image). */
264272 placementTries: number;
273+ /** Times the platform replaced the job's runner before its steps ran, and the job started again. */
274+ restarts: number;
265275 };
266276
267277 const toRun = (r: RunRow): RunInfo => ({
@@ -302,6 +312,7 @@
302312 runner: j.runner_info ? (JSON.parse(j.runner_info) as JobRunnerInfo) : null,
303313 carriedFrom: j.carried_from ?? null,
304314 placementTries: j.placement_tries ?? 0,
315+ restarts: j.restarts ?? 0,
305316 });
306317
307318 /**
@@ -1142,13 +1153,14 @@
11421153 }
11431154 this.sql.exec("UPDATE jobs SET fleet_held = 0 WHERE id = ?", jobId);
11441155 const token = `djt_${m.repoId}_${jobId}_${randomToken(24)}`;
1145- const container = `job-${jobId}`;
1156+ // A restarted job gets a new container: the replaced one may still be shutting down.
1157+ const container = job.restarts ? `job-${jobId}-r${job.restarts}` : `job-${jobId}`;
11461158 this.sql.exec("UPDATE jobs SET status = 'waiting', token_hash = ?, container = ? WHERE id = ?", await sha256(token), container, jobId);
11471159 await startCompute(this.env, { id: jobId, accountId: m.ownerId, repoId: m.repoId, kind: "job" }).catch((e: unknown) => console.error("compute usage failed", e));
11481160 this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
11491161 try {
11501162 const stub = this.env.RUNNER.get(this.env.RUNNER.idFromName(container));
1151- await stub.startJob({ repoId: m.repoId, jobId, token, apiUrl: this.env.PUBLIC_URL, egress: await egressFor(this.env, m.ownerId) });
1163+ await stub.startJob({ repoId: m.repoId, jobId, container, token, apiUrl: this.env.PUBLIC_URL, egress: await egressFor(this.env, m.ownerId) });
11521164 } catch (e) {
11531165 const message = (e as Error).message;
11541166 if (RETRYABLE_START.test(message)) return this.placementFailed(jobId, message);
@@ -1438,15 +1450,20 @@
14381450 await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: `The job exceeded its timeout of ${j.timeout_minutes} minutes.` } }, "timed_out");
14391451 }
14401452
1441- async containerStopped(jobId: string, exitCode: number | null): Promise<void> {
1453+ async containerStopped(jobId: string, exitCode: number | null, container?: string): Promise<void> {
14421454 const j = this.sql.exec<JobRow>("SELECT * FROM jobs WHERE id = ?", jobId).toArray()[0];
14431455 if (!j || j.status === "completed") return;
1456+ // A container the job no longer runs in (it was replaced, and the job restarted in another).
1457+ if (container && j.container !== container) return;
14441458 // Already back in line (placementFailed): a container given up on, which started after all,
14451459 // finds its token revoked and exits. The job's next placement starts a new one.
14461460 if (j.status === "queued") return;
14471461 // Stopped before the runner fetched the job, so no step ran: place it again. During a
14481462 // container rollout a new container can exit (or never get a machine) before it reports in.
14491463 if (j.status === "waiting" && !j.labels) return this.placementFailed(jobId, exitCode ? `the runner exited with code ${exitCode} before it started the job` : "the container stopped before the runner started");
1464+ // Stopped without reporting while a newer runner image is deployed than the one it ran on: a rollout replaced it.
1465+ const ran = j.runner_info ? (JSON.parse(j.runner_info) as JobRunnerInfo).image : undefined;
1466+ if (!j.labels && ran && this.env.RUNNER_IMAGE && ran !== this.env.RUNNER_IMAGE) return this.runnerReplaced(j);
14501467 await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: exitCode ? `The runner stopped unexpectedly (exit code ${exitCode}).` : "The runner's container stopped before the job finished, without an exit code (the platform restarted or replaced it)." } });
14511468 }
14521469
@@ -1967,10 +1984,61 @@
19671984 }
19681985
19691986 async runnerComplete(jobId: string, token: string, completion: JobCompletion): Promise<void> {
1970- await this.authJob(jobId, token);
1987+ const job = await this.authJob(jobId, token);
1988+ if (completion.replaced && completion.conclusion !== "success" && !job.labels) return this.runnerReplaced(job);
19711989 await this.completeJob(jobId, completion);
19721990 }
19731991
1992+ /**
1993+ * The platform replaced a hosted job's runner while the job ran (a deploy's container rollout).
1994+ * Before any step but checkout ran, the job starts again from the beginning on a new runner, as
1995+ * if it had just been queued, up to MAX_RUNNER_RESTARTS times. After that, or once a step ran
1996+ * whose effects running it again could repeat, the job fails and says why.
1997+ */
1998+ private async runnerReplaced(job: JobRow): Promise<void> {
1999+ const run = this.runRow(job.run_id);
2000+ const steps = JSON.parse(job.steps) as Array<StepReport & { index: number }>;
2001+ let defs: Job["steps"] = [];
2002+ try {
2003+ defs = (await this.expanded(run)).workflow.jobs[job.job_key]?.steps ?? [];
2004+ } catch (e) {
2005+ console.error("could not read the job's steps", e);
2006+ }
2007+ const ran = steps.find((s) => s.startedAt && !/^actions\/checkout@/.test(defs[s.index]?.uses ?? ""));
2008+ const restarts = (job.restarts ?? 0) + 1;
2009+ if (ran || restarts > MAX_RUNNER_RESTARTS) {
2010+ const why = ran
2011+ ? `while step "${ran.name}" ran. Diffinery doesn't start a job again by itself once its steps have run, since running them twice could repeat what they did`
2012+ : `before the job's steps ran, ${MAX_RUNNER_RESTARTS + 1} times`;
2013+ await this.completeJob(job.id, { conclusion: "failure", outputs: {}, failure: { message: `The platform replaced this job's runner with a new version ${why}. Re-run the job.` } });
2014+ return;
2015+ }
2016+ const t = now();
2017+ this.sql.exec(
2018+ "UPDATE jobs SET status = 'queued', container = NULL, token_hash = NULL, started_at = NULL, steps = '[]', outputs = '{}', failure = NULL, summary = NULL, runner_info = NULL, log_chars = 0, restarts = ? WHERE id = ?",
2019+ restarts,
2020+ job.id,
2021+ );
2022+ // The log starts over, with a line that says why (its seq keeps counting up, for live readers).
2023+ const text = `##[warning]The platform replaced this job's runner with a new version before the job's steps ran. Diffinery started the job again from the beginning on a new runner (restart ${restarts} of ${MAX_RUNNER_RESTARTS}).`;
2024+ const seq = (this.sql.exec<{ n: number | null }>("SELECT max(seq) AS n FROM logs WHERE job_id = ?", job.id).one().n ?? -1) + 1;
2025+ this.sql.exec("DELETE FROM logs WHERE job_id = ?", job.id);
2026+ this.sql.exec("INSERT INTO logs (job_id, seq, t, step, text) VALUES (?, ?, ?, -1, ?)", job.id, seq, t, text);
2027+ this.sql.exec("UPDATE jobs SET log_chars = ? WHERE id = ?", text.length, job.id);
2028+ await this.repo.runtime.sweepJob(job.id).catch((e) => console.error("runtime sweep failed", e));
2029+ await endCompute(this.env, job.id).catch((e: unknown) => console.error("compute usage failed", e));
2030+ this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId: job.id, detailsUrl: this.jobUrl(run, job.id) });
2031+ this.repo.broadcast(`job:${job.id}`, { action: "restarted" });
2032+ this.repo.broadcast(`run:${run.id}`, { action: "job", jobId: job.id });
2033+ try {
2034+ const instance = await this.env.RUN_WORKFLOW.get(run.id);
2035+ await instance.sendEvent({ type: `job-${job.id}`, payload: { retryInSeconds: 10 } });
2036+ } catch (e) {
2037+ console.error("could not ask the run's workflow to start the job again", e);
2038+ await this.completeJob(job.id, { conclusion: "failure", outputs: {}, failure: { message: "The platform replaced this job's runner with a new version, and Diffinery could not start the job again. Re-run the job." } });
2039+ }
2040+ }
2041+
19742042 private async completeJob(jobId: string, completion: JobCompletion, override?: "timed_out"): Promise<void> {
19752043 const job = this.jobRow(jobId);
19762044 if (job.status === "completed") return;
modifiedapps/diffinery/src/repo/schema.ts+2−0View file
@@ -199,6 +199,8 @@
199199 ["jobs", "placement_tries", "INTEGER NOT NULL DEFAULT 0"],
200200 // What the runner reported when it fetched the job (JSON RunnerInfo: image, location, instance), for investigations.
201201 ["jobs", "runner_info", "TEXT"],
202+ // Times the platform replaced a hosted job's runner before its steps ran and the job started again (repo/actions.ts MAX_RUNNER_RESTARTS).
203+ ["jobs", "restarts", "INTEGER NOT NULL DEFAULT 0"],
202204 // The GitHub import that wrote this row (null for everything else); a retry may only overwrite its own rows.
203205 ["issues", "imported", "TEXT"],
204206 ["pulls", "imported", "TEXT"],
modifiedapps/diffinery/src/runner/containers.ts+4−4View file
@@ -45,8 +45,8 @@
4545 // Jobs make no inbound requests, so activity-based sleep must not stop them.
4646 override sleepAfter = "6h";
4747
48- async startJob(input: { repoId: string; jobId: string; token: string; apiUrl: string; egress?: Egress }): Promise<void> {
49- await this.ctx.storage.put("job", { repoId: input.repoId, jobId: input.jobId });
48+ async startJob(input: { repoId: string; jobId: string; container?: string; token: string; apiUrl: string; egress?: Egress }): Promise<void> {
49+ await this.ctx.storage.put("job", { repoId: input.repoId, jobId: input.jobId, container: input.container });
5050 const egressEnv = await this.useEgress(input.egress ?? "open");
5151 await this.start({
5252 envVars: {
@@ -65,10 +65,10 @@
6565 }
6666
6767 override async onStop(params: StopParams): Promise<void> {
68- const job = await this.ctx.storage.get<{ repoId: string; jobId: string }>("job");
68+ const job = await this.ctx.storage.get<{ repoId: string; jobId: string; container?: string }>("job");
6969 if (!job) return;
7070 const repo = this.env.REPO.get(this.env.REPO.idFromName(job.repoId));
71- await repo.containerStopped(job.jobId, params.exitCode ?? null);
71+ await repo.containerStopped(job.jobId, params.exitCode ?? null, job.container);
7272 await this.ctx.storage.delete("job");
7373 }
7474
modifiedapps/diffinery/src/runner/workflow.ts+4−3View file
@@ -1,7 +1,7 @@
11 import { WorkflowEntrypoint, type WorkflowEvent, type WorkflowStep } from "cloudflare:workers";
22 import type { Env } from "../env.js";
33 import { graphFromLevels, runGraph, type JobNode } from "../repo/job-graph.js";
4-import { MAX_PLACEMENT_TRIES } from "../repo/actions.js";
4+import { MAX_PLACEMENT_TRIES, MAX_RUNNER_RESTARTS } from "../repo/actions.js";
55
66 export interface RunParams {
77 repoId: string;
@@ -51,7 +51,8 @@
5151 }
5252 }
5353 if (status === "completed" || status === "approval") return;
54- // A job whose container got no machine comes back with `retryInSeconds`: pause, start it again
54+ // A job whose container got no machine, or whose runner a rollout replaced before its steps ran
55+ // (repo/actions.ts runnerReplaced), comes back with `retryInSeconds`: pause, start it again
5556 // and keep waiting. The first round keeps its step names, so runs already in flight replay.
5657 for (let i = 0; ; i++) {
5758 const n = i ? ` ${i}` : "";
@@ -64,7 +65,7 @@
6465 return;
6566 }
6667 const pause = event.payload?.retryInSeconds;
67- if (!pause || i >= MAX_PLACEMENT_TRIES) return;
68+ if (!pause || i >= MAX_PLACEMENT_TRIES + MAX_RUNNER_RESTARTS) return;
6869 await step.sleep(`pause ${jobId} ${i + 1}`, `${pause} seconds`);
6970 await step.do(`relaunch ${jobId} ${i + 1}`, () => repo().relaunchJob(jobId));
7071 }
modifiedapps/diffinery/src/web/assets.ts+1−1View file
@@ -455,7 +455,7 @@
455455 if(mode==='log'){if(!String(m.topic).startsWith('job:'))return;
456456 if(a==='logs'){append(m.data.lines||[]);return}
457457 if(a==='step'&&m.data.step){stepState(m.data.step);return}
458- if(a==='completed'){setTimeout(()=>location.reload(),700)}return}
458+ if(a==='completed'||a==='restarted'){setTimeout(()=>location.reload(),700)}return}
459459 clearTimeout(timer);timer=setTimeout(refresh,350)}
460460 async function refresh(){const frags=$$('[data-fragment]');if(!frags.length){location.reload();return}
461461 try{const r=await fetch(location.href,{credentials:'same-origin'});if(!r.ok)return;const doc=new DOMParser().parseFromString(await r.text(),'text/html');
modifiedapps/diffinery/test/unit/actions-runtime.test.ts+97−1View file
@@ -1,7 +1,7 @@
11 import { describe, expect, it } from "vitest";
22 import { BodyTooLarge, STEP_OPS, batchParts, blockKey, chunkKey, compose, finalKey, listStaged, parseBlockList, parseRange, planCompose, putStream, runComposeStep, stagingPrefix, type BlobBucket, type ComposeLimits, type ComposeStep } from "../../src/lib/blob-store.js";
33 import { DatabaseSync } from "node:sqlite";
4-import { Actions, MAX_ACTIVE_RUNS, MAX_HOSTED_JOBS, MAX_PLACEMENT_TRIES, parseRunnerInfo, placementDelay, STALE_IMAGE_TRIES } from "../../src/repo/actions.js";
4+import { Actions, MAX_ACTIVE_RUNS, MAX_HOSTED_JOBS, MAX_PLACEMENT_TRIES, MAX_RUNNER_RESTARTS, parseRunnerInfo, placementDelay, STALE_IMAGE_TRIES } from "../../src/repo/actions.js";
55 import { testD1 } from "./stubs/d1.js";
66 import { DEFAULT_ARTIFACT_QUOTA, Runtime, artifactFileName, artifactObjectKeys } from "../../src/repo/runtime.js";
77 import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
@@ -818,6 +818,102 @@
818818 // Self-hosted runners run any image.
819819 await expect(refuse("s1", "my-image")).resolves.toBeUndefined();
820820 });
821+
822+ it("a job whose runner a rollout replaced starts again on a new runner before its steps ran, and fails after", async () => {
823+ const sql = sqlStorage();
824+ sql.exec(REPO_SCHEMA);
825+ migrateRepoSchema(sql as unknown as SqlStorage);
826+ const events: { type: string; payload: Record<string, unknown> }[] = [];
827+ const repo = {
828+ sql,
829+ meta: { repoId: "r", owner: "acme", ownerId: "acme", name: "app", defaultBranch: "main" },
830+ upsertCheck: () => {},
831+ broadcast: () => {},
832+ deploys: { jobCompleted: () => {} },
833+ runtime: { sweepJob: async () => {}, grantFor: () => ({ role: "read", permissions: {} }) },
834+ } as unknown as RepoDO;
835+ const DB = testD1();
836+ DB.sqlite.prepare("INSERT INTO accounts (id, login, kind, status, created_at) VALUES ('acme', 'acme', 'org', 'active', ?)").run(new Date().toISOString());
837+ const started: { container: string; token: string }[] = [];
838+ const env = {
839+ DB,
840+ PUBLIC_URL: "https://d.test",
841+ BUCKET: memoryBucket().bucket,
842+ RUNNER_IMAGE: "diffinery-runner:new",
843+ RUNNER: { idFromName: (n: string) => n, get: () => ({ startJob: async (i: { container: string; token: string }) => void started.push(i), stopJob: async () => {} }) },
844+ RUN_WORKFLOW: { get: async () => ({ sendEvent: async (e: { type: string; payload: Record<string, unknown> }) => void events.push(e) }) },
845+ } as unknown as Env;
846+ const actions = new Actions(repo, env);
847+ // The job's steps: checkout, then one whose effects running it twice would repeat.
848+ (actions as unknown as { expanded: () => Promise<unknown> }).expanded = async () => {
849+ const steps = [{ uses: "actions/checkout@v4" }, { run: "make deploy" }];
850+ return { workflow: { jobs: { k: { steps }, k2: { steps }, k3: { steps } } } };
851+ };
852+ const t = new Date().toISOString();
853+ sql.exec("INSERT INTO runs (id, run_number, workflow_path, workflow_name, event, ref, sha, actor, status, created_at, updated_at, event_payload) VALUES ('run1', 1, 'w.yml', 'w', 'push', 'refs/heads/main', 'abc', '{}', 'in_progress', ?, ?, '{}')", t, t);
854+ sql.exec("INSERT INTO jobs (id, run_id, job_key, name, status) VALUES ('j1', 'run1', 'k', 'k', 'queued'), ('j2', 'run1', 'k2', 'k2', 'queued')");
855+ const launch = (id: string) => (actions as unknown as { launch(id: string): Promise<void> }).launch(id);
856+ const job = (id: string) => sql.exec("SELECT status, conclusion, container, token_hash, steps, restarts, failure FROM jobs WHERE id = ?", id).toArray()[0];
857+ const logs = (id: string) => sql.exec("SELECT seq, step, text FROM logs WHERE job_id = ? ORDER BY seq", id).toArray();
858+ // The runner fetched the job on `image` and reported these steps as started.
859+ const running = (id: string, image: string, steps: number[]) => {
860+ const reports = steps.map((index) => ({ index, name: index ? "Deploy" : "Run actions/checkout@v4", status: "in_progress", startedAt: t }));
861+ sql.exec("UPDATE jobs SET status = 'in_progress', started_at = ?, runner_info = ?, steps = ? WHERE id = ?", t, JSON.stringify({ image }), JSON.stringify(reports), id);
862+ sql.exec("INSERT INTO logs (job_id, seq, t, step, text) VALUES (?, 0, ?, -1, 'set up'), (?, 1, ?, 0, 'checking out')", id, t, id, t);
863+ };
864+ const replaced = { conclusion: "cancelled" as const, outputs: {}, replaced: true };
865+
866+ // The runner reports the replacement during checkout: the job goes back in line, its log starts over with a line that says why.
867+ await launch("j1");
868+ const first = started.at(-1)!;
869+ expect(first.container).toBe("job-j1");
870+ running("j1", "diffinery-runner:old", [0]);
871+ await actions.runnerComplete("j1", first.token, replaced);
872+ expect(job("j1")).toMatchObject({ status: "queued", conclusion: null, container: null, token_hash: null, steps: "[]", restarts: 1 });
873+ expect(events.at(-1)).toEqual({ type: "job-j1", payload: { retryInSeconds: 10 } });
874+ expect(logs("j1")).toEqual([{ seq: 2, step: -1, text: expect.stringMatching(/replaced this job's runner.*started the job again from the beginning on a new runner \(restart 1 of 2\)/) }]);
875+ // The replaced runner can't report any more, and its container stopping changes nothing.
876+ await expect(actions.runnerComplete("j1", first.token, replaced)).rejects.toMatchObject({ status: 401 });
877+ await actions.containerStopped("j1", 143, "job-j1");
878+ expect(job("j1")).toMatchObject({ status: "queued", restarts: 1 });
879+
880+ // It starts in a new container; the old one stopping late doesn't touch it.
881+ await actions.relaunchJob("j1");
882+ expect(started.at(-1)!.container).toBe("job-j1-r1");
883+ await actions.containerStopped("j1", 143, "job-j1");
884+ expect(job("j1")).toMatchObject({ status: "waiting", container: "job-j1-r1" });
885+
886+ // A container on a replaced image that stops without reporting counts the same.
887+ running("j1", "diffinery-runner:old", []);
888+ await actions.containerStopped("j1", 143, "job-j1-r1");
889+ expect(job("j1")).toMatchObject({ status: "queued", restarts: 2 });
890+
891+ // Past MAX_RUNNER_RESTARTS the job fails, and says why.
892+ await actions.relaunchJob("j1");
893+ expect(started.at(-1)!.container).toBe("job-j1-r2");
894+ running("j1", "diffinery-runner:old", []);
895+ await actions.runnerComplete("j1", started.at(-1)!.token, replaced);
896+ expect(job("j1")).toMatchObject({ status: "completed", conclusion: "failure", restarts: MAX_RUNNER_RESTARTS });
897+ expect(JSON.parse(job("j1")!.failure as string).message).toMatch(/replaced this job's runner with a new version before the job's steps ran, 3 times\. Re-run the job\./);
898+
899+ // Once a step other than checkout ran, the job isn't started again: it fails and names the step.
900+ await launch("j2");
901+ running("j2", "diffinery-runner:old", [0, 1]);
902+ await actions.runnerComplete("j2", started.at(-1)!.token, replaced);
903+ expect(job("j2")).toMatchObject({ status: "completed", conclusion: "failure", restarts: 0 });
904+ expect(JSON.parse(job("j2")!.failure as string).message).toMatch(/while step "Deploy" ran\. Diffinery doesn't start a job again by itself/);
905+
906+ // A job that finished despite the replacement, or a runner on the deployed image that stops, completes as before.
907+ sql.exec("INSERT INTO jobs (id, run_id, job_key, name, status) VALUES ('j3', 'run1', 'k3', 'k3', 'queued'), ('j4', 'run1', 'k4', 'k4', 'queued')");
908+ await launch("j3");
909+ running("j3", "diffinery-runner:old", [0, 1]);
910+ await actions.runnerComplete("j3", started.at(-1)!.token, { conclusion: "success", outputs: {}, replaced: true });
911+ expect(job("j3")).toMatchObject({ status: "completed", conclusion: "success" });
912+ await launch("j4");
913+ running("j4", "diffinery-runner:new", []);
914+ await actions.containerStopped("j4", 137, "job-j4");
915+ expect(job("j4")).toMatchObject({ status: "completed", conclusion: "failure", restarts: 0 });
916+ });
821917 });
822918
823919 describe("runner info", () => {
modifiedapps/diffinery/wrangler.jsonc+5−1View file
@@ -77,7 +77,11 @@
7777 // A deploy that changes the image rolls the running containers: once a container has been up
7878 // longer than rollout_active_grace_period seconds, it gets SIGTERM and is killed 15 minutes later.
7979 // Runner and import containers are covered for their whole life (a job is capped at 6 hours, an
80- // import transfer too), so a deploy never cuts one short; they pick up the new image next start.
80+ // import transfer too), so a deploy shouldn't cut one short; they pick up the new image next start.
81+ // The platform doesn't guarantee it: a job that started on an old-image container during a rollout
82+ // was replaced minutes in. So a runner on a replaced image is turned away before it runs anything
83+ // (repo/actions.ts STALE_IMAGE_TRIES), and a job whose runner is replaced before its steps ran
84+ // starts again on a new one (MAX_RUNNER_RESTARTS).
8185 // The rollout is one step (rollout_step_percentage 100): with the default [10, 100], the first
8286 // step still starts 90% of new containers on the old image, and the next step replaced such a
8387 // job's container seconds after it started. The rollout's SIGTERM reaches every process in the
modifiedcontainers/runner/Dockerfile+3−2View file
@@ -22,10 +22,11 @@
2222 PIP_BREAK_SYSTEM_PACKAGES=1
2323
2424 # The -dev libraries are what ruby/setup-ruby's prebuilt Rubies link against (psych needs
25-# libyaml) and what gems' native extensions build with, as on hosted runners.
25+# libyaml) and what gems' native extensions build with, as on hosted runners. Jobs run as root,
26+# and sudo is there (passwordless, as root) so `sudo apt-get install ...` works as on GitHub.
2627 RUN apt-get update \
2728 && apt-get install -y --no-install-recommends \
28- ca-certificates curl git git-lfs jq zip unzip xz-utils tar gzip zstd \
29+ ca-certificates curl git git-lfs jq zip unzip xz-utils tar gzip zstd sudo \
2930 build-essential pkg-config openssh-client gnupg \
3031 python3 python3-pip python3-venv python-is-python3 \
3132 libyaml-dev libssl-dev libffi-dev zlib1g-dev libreadline-dev libgmp-dev \
modifiedcontainers/runner/src/agent.ts+1−1View file
@@ -47,7 +47,7 @@
4747 if (!o.drainOnSigtermMs || draining) return onSignal("SIGTERM");
4848 const minutes = Math.round(o.drainOnSigtermMs / 60_000);
4949 runner.replacing = true;
50- runner.logger.line(-1, `##[warning]This runner is being replaced by a new version. The job keeps running for up to ${minutes} minutes before it is cancelled, but a step the platform's stop signal ends fails.`);
50+ runner.logger.line(-1, `##[warning]This runner is being replaced by a new version. The job keeps running for up to ${minutes} minutes; if the platform stops a step or the time runs out, Diffinery starts the job again on a new runner.`);
5151 draining = setTimeout(() => onSignal("SIGTERM"), o.drainOnSigtermMs);
5252 draining.unref();
5353 };
modifiedcontainers/runner/src/job.ts+5−2View file
@@ -214,6 +214,7 @@
214214 if (summary) completion.summary = this.masker.mask(summary);
215215 if (conclusion === "failure" && this.firstFailure) completion.failure = { ...this.firstFailure, message: this.masker.mask(this.firstFailure.message) };
216216 if (conclusion === "cancelled") completion.failure = { message: "The job was cancelled." };
217+ if (this.replacing && conclusion !== "success") completion.replaced = true;
217218 this.logger.line(-1, `Job completed with conclusion: ${conclusion}`);
218219 return completion;
219220 }
@@ -636,10 +637,12 @@
636637 });
637638 await this.applyFileCommands(files, o.io);
638639 if (res.signal && this.replacing && !this.stopping) {
639- // A rollout's SIGTERM reaches every process in the container, not only the agent.
640- const msg = "The platform stopped this step's processes while replacing the runner with a new version. Re-run the job.";
640+ // A rollout's SIGTERM reaches every process in the container, not only the agent. The job
641+ // can't succeed now, so stop it and report at once: Diffinery starts it again on a new runner.
642+ const msg = "The platform stopped this step's processes while replacing the runner with a new version.";
641643 this.logger.line(o.io.logIndex, `##[error]${msg}`);
642644 this.addAnnotation(o.io.logIndex, { level: "error", message: msg });
645+ this.cancel("cancelled");
643646 }
644647 if (res.timedOut) {
645648 const mins = (o.timeoutMs ?? 0) / 60000;
modifiedcontainers/runner/test/job.test.ts+10−6View file
@@ -812,6 +812,7 @@
812812 drain.child.kill("SIGTERM");
813813 expect(await drain.exited).toBe(0);
814814 expect(api.completion).toMatchObject({ conclusion: "success" });
815+ expect(api.completion).not.toHaveProperty("replaced");
815816 expect(api.logText()).toContain("being replaced by a new version");
816817 expect(api.logText(1)).toContain("after");
817818
@@ -824,7 +825,7 @@
824825 stop.child.kill("SIGTERM");
825826 expect(await stop.exited).toBe(1);
826827 expect(Date.now() - t0).toBeLessThan(15_000);
827- expect(api.completion).toMatchObject({ conclusion: "cancelled" });
828+ expect(api.completion).toMatchObject({ conclusion: "cancelled", replaced: true });
828829 expect(api.lastReport(0)?.conclusion).toBe("cancelled");
829830 expect(api.lastReport(1)?.conclusion).toBe("cancelled");
830831
@@ -836,13 +837,15 @@
836837 expect(await cancel.exited).toBe(1);
837838 expect(Date.now() - t1).toBeLessThan(15_000);
838839 expect(api.completion).toMatchObject({ conclusion: "cancelled" });
840+ expect(api.completion).not.toHaveProperty("replaced");
839841 expect(api.logText()).not.toContain("being replaced");
840842
841843 // A rollout's SIGTERM reaches every process in the container, steps included: the
842- // step fails with a message that names the rollout, and the job finishes without
843- // waiting out the drain.
844+ // step fails with a message that names the rollout, the rest of the job is cancelled
845+ // without waiting out the drain, and the result says the runner was replaced, so
846+ // Diffinery can start the job again on a new runner.
844847 const t2 = Date.now();
845- const rollout = startJob("job-6", [{ run: "echo started $$; sleep 60" }]);
848+ const rollout = startJob("job-6", [{ run: "echo started $$; sleep 60" }, { run: "echo never" }]);
846849 while (!/started \d+/.test(api.logText()) && Date.now() - t2 < 15_000) await new Promise((r) => setTimeout(r, 50));
847850 const pid = Number(/started (\d+)/.exec(api.logText())?.[1]);
848851 rollout.child.kill("SIGTERM");
@@ -850,8 +853,9 @@
850853 process.kill(-pid, "SIGTERM");
851854 expect(await rollout.exited).toBe(1);
852855 expect(Date.now() - t2).toBeLessThan(15_000);
853- expect(api.completion).toMatchObject({ conclusion: "failure" });
856+ expect(api.completion).toMatchObject({ conclusion: "cancelled", replaced: true });
857+ expect(api.lastReport(0)?.conclusion).toBe("cancelled");
858+ expect(api.lastReport(1)?.conclusion).toBe("cancelled");
854859 expect(api.logText()).toContain("The platform stopped this step's processes while replacing the runner");
855- expect(api.logText()).toContain("Process completed with exit code 143.");
856860 }, 60_000);
857861 });
modifiedpackages/shared/src/runner.ts+5−0View file
@@ -123,6 +123,11 @@
123123 summary?: string;
124124 /** Short machine-readable reason when the job failed (for agents). */
125125 failure?: { step?: number; message: string };
126+ /**
127+ * Hosted runners: the platform began replacing the runner's container (a rollout's SIGTERM)
128+ * and the job didn't succeed, so Diffinery may start it again on a new runner.
129+ */
130+ replaced?: boolean;
126131 };
127132
128133 /** Runner API paths, relative to `${apiUrl}/runner/jobs/${jobId}`. */