Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Container rollouts: new containers get the new image at once; Deploy waits for it (#127)

* Container rollouts: new containers get the new image at once; Deploy waits for it

A main sync failed when the Live test's publish job started on the old runner image
during a container rollout and was replaced 12 seconds later. The rollout's SIGTERM
reached the step's own processes (bash and the docker CLI), not only the agent, so
the build died even though the agent was draining.

- wrangler.jsonc: rollout_step_percentage 100 for every container class. With the
  default [10, 100], the first step still schedules 90% of new containers on the old
  image; running containers stay protected by rollout_active_grace_period.
- Deploy waits (up to 10 minutes) for the rollout of its image to reach its last
  step before it finishes, so the Live test's jobs start on the new image.
- A step that a rollout's stop signal ends fails with a message naming the rollout,
  and a step killed by a signal reports 128 + the signal number (143 for SIGTERM).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ruyx3Qr4jYrZEpPi466deb

* Deploy notes that its own job, on an old-image container, outlived the rollout

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ruyx3Qr4jYrZEpPi466deb

---------

Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent 5619cf2commit 93ef0c8bfc6ff6ea65b67f13cd2c281b5f8aff46Browse files
11 files changed+90−8
Files (11)
modified.github/workflows/deploy.yml+6−0View file
@@ -67,6 +67,12 @@
6767 working-directory: apps/diffinery
6868 run: pnpm exec wrangler deploy -c wrangler.deploy.jsonc
6969
70+ - name: Wait for new containers to get the new image
71+ # The Live test follows this workflow, and its jobs must not start on the old image.
72+ # This job runs on an old-image container, so finishing after the rollout's last step
73+ # also shows that a deploy doesn't stop a running job.
74+ run: scripts/wait-container-rollout.sh
75+
7076 - name: Generate Worker secrets (first deploy only)
7177 run: scripts/ensure-secrets.sh
7278
modifiedapps/diffinery/docs/actions/overview.md+1−1View file
@@ -431,7 +431,7 @@
431431 | A self-hosted job stays `waiting` | No runner with all the job's labels is polling. Start `diffinery runner` with those labels. After 6 hours the job times out. |
432432 | `Could not start a runner: ...` | The job's container failed to start. Re-run the run. |
433433 | `The runner stopped unexpectedly (exit code <n>).` | The job's container exited before reporting a result. Re-run the run; if it repeats, check that the job doesn't exhaust memory or disk. |
434-| `This runner is being replaced by a new version. The job keeps running for up to 13 minutes before it is cancelled.` | The platform is replacing this job's container. Deploys leave running jobs alone, so this is rare. Jobs that finish within 13 minutes are unaffected; longer ones are cancelled and need a re-run. |
434+| `This runner is being replaced by a new version. The job keeps running for up to 13 minutes before it is cancelled, but a step the platform's stop signal ends fails.` | The platform is replacing this job's container. Deploys leave running jobs alone, so this is rare. The platform's stop signal reaches the running step as well, which then fails with "The platform stopped this step's processes while replacing the runner with a new version." Re-run the job. |
435435 | `Error evaluating 'if': ...` | The job's `if:` expression failed at runtime. See [Expressions](/docs/actions/expressions). |
436436 | A step can't reach a secret | Secrets are configured in the repository or organization settings. See [Secrets and variables](/docs/actions/secrets-and-variables). |
437437 | `git push` from a job fails | The job's `GITHUB_TOKEN` needs `contents: write` (new repositories default to read-only tokens), can't change workflow files, and follows branch protection. See [GITHUB_TOKEN permissions](/docs/actions/workflow-permissions). |
modifiedapps/diffinery/docs/actions/runners.md+1−1View file
@@ -142,7 +142,7 @@
142142 | Matrix size | 256 jobs per matrix | The job fails with "A matrix may produce at most 256 jobs." |
143143 | Log line length | 16,384 characters | Longer lines are cut and end with `…` |
144144 | Log size | 16 MiB of text per job | The log ends with "The log is cut off here: a job's log is limited to 16 MiB of text." Later lines are dropped; the job keeps running. |
145-| Platform updates | A deploy doesn't replace the container of a running job: running jobs finish on the version they started with. If the platform replaces a container anyway, the job keeps going for up to 13 minutes | The log shows "This runner is being replaced by a new version. The job keeps running for up to 13 minutes before it is cancelled." |
145+| Platform updates | A deploy doesn't replace the container of a running job: running jobs finish on the version they started with. If the platform replaces a container anyway, its stop signal reaches the running step too, so that step fails and the job needs a re-run | The log shows "This runner is being replaced by a new version" and, on the step, "The platform stopped this step's processes while replacing the runner with a new version." |
146146
147147 If the container stops unexpectedly (for example it runs out of memory), the job fails with `The runner stopped unexpectedly (exit code N).`
148148
modifiedapps/diffinery/docs/reference/limits.md+1−0View file
@@ -358,6 +358,7 @@
358358 | Sandbox containers | `standard-1`, up to 50 instances | `containers[SandboxContainer]`. |
359359 | Import containers | `standard-4`, up to 20 instances | `containers[ImportContainer]`: one container per Git transfer of an import. |
360360 | Container rollout grace period | 23,400 seconds for runners, 25,200 for imports, 3,600 for sandboxes | `rollout_active_grace_period`. A deploy doesn't replace a container younger than this, so running jobs (6 hours at most) and import transfers finish first. |
361+| Container rollout steps | One step of 100% | `rollout_step_percentage`. Containers started after a deploy get the new image at once, so a new job doesn't start on the old one and get replaced. |
361362 | Event queue | Batches of up to 50 messages (2 seconds), 8 retries | `diffinery-events`, dead-letter queue `diffinery-events-dlq`. |
362363 | Mail queue | Batches of up to 20 messages (2 seconds), 8 retries | `diffinery-mail`, dead-letter queue `diffinery-mail-dlq`. The consumer itself gives up after 6 attempts. |
363364 | Workflow scheduler | Every 5 minutes | `SchedulerDO`'s own alarm (`SCHEDULE_TICK_MINUTES`, 1 to 60) starts scheduled workflows, up to 16 repositories at a time. |
modifiedapps/diffinery/docs/self-hosting/configuration.md+1−1View file
@@ -117,7 +117,7 @@
117117 | `observability` | Enabled, `head_sampling_rate: 1` | Every request is logged to Workers Logs. |
118118 | `upload_source_maps` | `true` | Readable stack traces in the logs. |
119119 | `limits.cpu_ms` | `300000` | Up to 5 minutes of CPU time per request (large pushes, diffs and merges). |
120-| `containers` | `RunnerContainer`: `standard-2`, `max_instances: 100`; `SandboxContainer`: `standard-1`, `max_instances: 50`; `ImportContainer`: `standard-4`, `max_instances: 20`; `rollout_active_grace_period`: 23400 for runners and 25200 for imports (longer than a job or transfer can run, so a deploy never replaces a running one), 3600 for sandboxes | All three build `containers/runner/Dockerfile` with `containers/runner` as context. Raise `max_instances` for more concurrent jobs, sandboxes or import transfers. |
120+| `containers` | `RunnerContainer`: `standard-2`, `max_instances: 100`; `SandboxContainer`: `standard-1`, `max_instances: 50`; `ImportContainer`: `standard-4`, `max_instances: 20`; `rollout_active_grace_period`: 23400 for runners and 25200 for imports (longer than a job or transfer can run, so a deploy never replaces a running one), 3600 for sandboxes; `rollout_step_percentage: 100`, so containers started after a deploy get the new image | All three build `containers/runner/Dockerfile` with `containers/runner` as context. Raise `max_instances` for more concurrent jobs, sandboxes or import transfers. |
121121 | `triggers.crons` | `["*/5 * * * *"]` | Token expiry reminders, email log pruning, registry cleanup, cancelling draft imports older than 7 days, and setting the workflow scheduler's alarm if it isn't set. Scheduled workflows run from that alarm, not from the cron trigger. |
122122
123123 ## The dev environment
modifiedapps/diffinery/wrangler.jsonc+7−3View file
@@ -74,11 +74,15 @@
7474 // longer than rollout_active_grace_period seconds, it gets SIGTERM and is killed 15 minutes later.
7575 // Runner and import containers are covered for their whole life (a job is capped at 6 hours, an
7676 // import transfer too), so a deploy never cuts one short; they pick up the new image next start.
77+ // The rollout is one step (rollout_step_percentage 100): with the default [10, 100], the first
78+ // step still starts 90% of new containers on the old image, and the next step replaced such a
79+ // job's container seconds after it started. The rollout's SIGTERM reaches every process in the
80+ // container, so the job's steps died with it.
7781 "containers": [
78- { "class_name": "RunnerContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-2", "max_instances": 100, "rollout_active_grace_period": 23400 },
79- { "class_name": "SandboxContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-1", "max_instances": 50, "rollout_active_grace_period": 3600 },
82+ { "class_name": "RunnerContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-2", "max_instances": 100, "rollout_active_grace_period": 23400, "rollout_step_percentage": 100 },
83+ { "class_name": "SandboxContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-1", "max_instances": 50, "rollout_active_grace_period": 3600, "rollout_step_percentage": 100 },
8084 // GitHub imports: a mirror clone of a large repository needs disk and memory.
81- { "class_name": "ImportContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-4", "max_instances": 20, "rollout_active_grace_period": 25200 }
85+ { "class_name": "ImportContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-4", "max_instances": 20, "rollout_active_grace_period": 25200, "rollout_step_percentage": 100 }
8286 ],
8387 // Housekeeping, and a check that the workflow scheduler's alarm is set (src/scheduler.ts).
8488 "triggers": { "crons": ["*/5 * * * *"] },
modifiedcontainers/runner/src/agent.ts+2−1View file
@@ -46,7 +46,8 @@
4646 const sigterm = () => {
4747 if (!o.drainOnSigtermMs || draining) return onSignal("SIGTERM");
4848 const minutes = Math.round(o.drainOnSigtermMs / 60_000);
49- runner.logger.line(-1, `##[warning]This runner is being replaced by a new version. The job keeps running for up to ${minutes} minutes before it is cancelled.`);
49+ runner.replacing = true;
50+ runner.logger.line(-1, `##[warning]This runner is being replaced by a new version. The job keeps running for up to ${minutes} minutes before it is cancelled, but a step the platform's stop signal ends fails.`);
5051 draining = setTimeout(() => onSignal("SIGTERM"), o.drainOnSigtermMs);
5152 draining.unref();
5253 };
modifiedcontainers/runner/src/job.ts+8−0View file
@@ -95,6 +95,8 @@
9595 readonly runtime: RuntimeClient | null;
9696
9797 private stopping: null | "cancelled" | "timeout" = null;
98+ /** Set when the platform signals that it is replacing this runner (a container rollout). */
99+ replacing = false;
98100 private firstFailure: { step?: number; message: string } | null = null;
99101 private githubCtx: Record<string, unknown> = {};
100102 private defaultEnv: Record<string, string> = {};
@@ -631,6 +633,12 @@
631633 },
632634 });
633635 await this.applyFileCommands(files, o.io);
636+ if (res.signal && this.replacing && !this.stopping) {
637+ // A rollout's SIGTERM reaches every process in the container, not only the agent.
638+ const msg = "The platform stopped this step's processes while replacing the runner with a new version. Re-run the job.";
639+ this.logger.line(o.io.logIndex, `##[error]${msg}`);
640+ this.addAnnotation(o.io.logIndex, { level: "error", message: msg });
641+ }
634642 if (res.timedOut) {
635643 const mins = (o.timeoutMs ?? 0) / 60000;
636644 const msg = `The ${o.timeoutLabel ?? "step"} has timed out after ${Number(mins.toFixed(2))} minutes.`;
modifiedcontainers/runner/src/util.ts+2−1View file
@@ -2,6 +2,7 @@
22 import { createHash } from "node:crypto";
33 import { existsSync, statSync, accessSync, readdirSync, readFileSync, constants as fsc } from "node:fs";
44 import path from "node:path";
5+import { constants } from "node:os";
56
67 export const sleep = (ms: number) => new Promise<void>((r) => setTimeout(r, ms));
78
@@ -194,7 +195,7 @@
194195 resolve({ exitCode: 127, signal: null, stdout, stderr: stderr + msg, timedOut, stdoutTruncated, stderrTruncated });
195196 return;
196197 }
197- resolve({ exitCode: code ?? (signal ? 128 : 1), signal, stdout, stderr, timedOut, stdoutTruncated, stderrTruncated });
198+ resolve({ exitCode: code ?? (signal ? 128 + (constants.signals[signal] ?? 0) : 1), signal, stdout, stderr, timedOut, stdoutTruncated, stderrTruncated });
198199 });
199200 });
200201 }
modifiedcontainers/runner/test/job.test.ts+16−0View file
@@ -831,5 +831,21 @@
831831 expect(Date.now() - t1).toBeLessThan(15_000);
832832 expect(api.completion).toMatchObject({ conclusion: "cancelled" });
833833 expect(api.logText()).not.toContain("being replaced");
834+
835+ // A rollout's SIGTERM reaches every process in the container, steps included: the
836+ // step fails with a message that names the rollout, and the job finishes without
837+ // waiting out the drain.
838+ const t2 = Date.now();
839+ const rollout = startJob("job-6", [{ run: "echo started $$; sleep 60" }]);
840+ while (!/started \d+/.test(api.logText()) && Date.now() - t2 < 15_000) await new Promise((r) => setTimeout(r, 50));
841+ const pid = Number(/started (\d+)/.exec(api.logText())?.[1]);
842+ rollout.child.kill("SIGTERM");
843+ await new Promise((r) => setTimeout(r, 200));
844+ process.kill(-pid, "SIGTERM");
845+ expect(await rollout.exited).toBe(1);
846+ expect(Date.now() - t2).toBeLessThan(15_000);
847+ expect(api.completion).toMatchObject({ conclusion: "failure" });
848+ expect(api.logText()).toContain("The platform stopped this step's processes while replacing the runner");
849+ expect(api.logText()).toContain("Process completed with exit code 143.");
834850 }, 60_000);
835851 });
addedscripts/wait-container-rollout.sh+45−0View file
@@ -1,0 +1,45 @@
1+#!/usr/bin/env bash
2+# Waits after `wrangler deploy` until the container rollouts it started reach their last step,
3+# when every new container starts on the new image. `wrangler deploy` returns as soon as a
4+# rollout starts, and a job started before its last step can land on the old image and be
5+# replaced (with its steps) moments later. Running containers keep the old image until they
6+# exit (rollout_active_grace_period), so the rollouts don't need to complete.
7+#
8+# Run from the repository root with CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID and
9+# RUNNER_IMAGE (scripts/runner-image.sh) set. Gives up with a warning after
10+# ROLLOUT_WAIT_MINUTES (default 10) rather than failing the deploy.
11+set -euo pipefail
12+: "${CLOUDFLARE_API_TOKEN:?}" "${CLOUDFLARE_ACCOUNT_ID:?}" "${RUNNER_IMAGE:?RUNNER_IMAGE is required (scripts/runner-image.sh)}"
13+api() { curl -fsS --retry 4 --retry-all-errors --retry-delay 3 "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/containers$1" -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"; }
14+deadline=$(( $(date +%s) + ${ROLLOUT_WAIT_MINUTES:-10} * 60 ))
15+last=""
16+while :; do
17+ if ! apps=$(api /applications); then
18+ echo "::warning::could not list the container applications, so the deploy doesn't wait for the rollout"
19+ exit 0
20+ fi
21+ pending=""
22+ states=""
23+ for row in $(jq -r '.result[] | select(.active_rollout_id != null) | "\(.id)/\(.active_rollout_id)/\(.name)"' <<<"$apps"); do
24+ IFS=/ read -r app rollout name <<<"$row"
25+ r=$(api "/applications/$app/rollouts/$rollout" | jq -c .result) || continue
26+ # Only the rollouts of this deploy's image (matched by its tag, a hash of the runner).
27+ [[ "$(jq -r '.target_configuration.image // ""' <<<"$r")" == *":${RUNNER_IMAGE##*:}" ]] || continue
28+ state=$(jq -r '"\(.status), step \((.progress.current_step // 0) + 1) of \(.progress.total_steps // (.steps | length)) \((.steps // []) | last | .status // ""), \(.progress.updated_instances // "?")/\(.progress.total_instances // "?") instances updated"' <<<"$r")
29+ states+="$name: $state; "
30+ # Done once the rollout ended, or its last step is under way (new containers get the new image).
31+ jq -e '(.status | IN("completed", "reverted", "replaced")) or ((.steps // []) | length > 0 and (last | .status | IN("progressing", "completed")))' <<<"$r" >/dev/null || pending+="$name "
32+ done
33+ [ "$states" = "$last" ] || { echo "${states:-no container rollout for $RUNNER_IMAGE}"; last=$states; }
34+ if [ -z "$pending" ]; then
35+ # This job's container started on the old image before the deploy; it is still running,
36+ # which shows a rollout leaves running jobs alone (rollout_active_grace_period).
37+ [ -z "$states" ] || echo "the rollout reached its last step and this job, on an old-image container, is still running"
38+ exit 0
39+ fi
40+ if [ "$(date +%s)" -ge "$deadline" ]; then
41+ echo "::warning::the container rollout of ${pending% } has not reached its last step after ${ROLLOUT_WAIT_MINUTES:-10} minutes; new jobs may still start on the old image"
42+ exit 0
43+ fi
44+ sleep 10
45+done