Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Jobs whose container stops before the runner fetched them are placed again; jobs record their runner (#135)

During #127's deploy two cors matrix legs failed with "The runner stopped unexpectedly
(exit code 2)" without running a step. A container start that returned "no container
instance" had put the job back in line and revoked its token, but the container
started anyway; its runner got a 401 fetching the job, exited with code 2, and that
stop failed the job that was waiting to be placed again.

- containerStopped ignores a stop for a job already back in line, and places a job
  again when its container stops (with or without an exit code) before the runner
  fetched it, since no step ran. MAX_PLACEMENT_TRIES still caps it.
- The runner sends its image tag (written next to the agent by runner-image.sh),
  Cloudflare location and container id with the job fetch; the job stores them and
  the API returns them as `runner` (with `runner_name`), and the log's first lines
  name them.
- The rollout wait counts started steps from the steps list ("step 1 of 1"; the
  API's current_step is 1-based in practice).


Claude-Session: https://claude.ai/code/session_01Ruyx3Qr4jYrZEpPi466deb

Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent 7a4c6b2commit 0858d861cf3a63f4e4a2484f16e36f3f5741342fBrowse files
15 files changed+99−17
Files (15)
modifiedapps/diffinery/docs/actions/runners.md+3−1View file
@@ -144,8 +144,10 @@
144144 | Log size | 16 MiB of text per job | The log ends with "The log is cut off here: a job's log is limited to 16 MiB of text." Later lines are dropped; the job keeps running. |
145145 | Platform updates | A deploy doesn't replace the container of a running job: running jobs finish on the version they started with. If the platform replaces a container anyway, its stop signal reaches the running step too, so that step fails and the job needs a re-run | The log shows "This runner is being replaced by a new version" and, on the step, "The platform stopped this step's processes while replacing the runner with a new version." |
146146
147-If the container stops unexpectedly (for example it runs out of memory), the job fails with `The runner stopped unexpectedly (exit code N).`
147+If the container stops unexpectedly (for example it runs out of memory), the job fails with `The runner stopped unexpectedly (exit code N).` A container that stops before its runner fetched the job is replaced instead, since no step ran.
148148
149+The log's first lines name where the job ran, for example `Runner: image diffinery-runner:96dbbe31790f, in fra06, EEUR, container 6f1c…`. The jobs API returns the same as `runner` (`image`, `location`, `instance`), next to `runner_name`.
150+
149151 ### Windows and macOS
150152
151153 Cloudflare Containers run Linux only, so Diffinery has no Windows or macOS hosted runners. `runs-on: windows-latest` or `runs-on: macos-latest` fails with the "No Diffinery runner matches runs-on" message. For macOS builds, Windows-specific tooling or GPUs, use a [self-hosted runner](#self-hosted-runners) on your own machine with `runs-on: [self-hosted, ...]`.
modifiedapps/diffinery/docs/reference/limits.md+1−1View file
@@ -202,7 +202,7 @@
202202 | Hosted runner size | Cloudflare Containers `standard-2` | Each `runs-on: ubuntu-latest` job runs in its own container of this instance type. |
203203 | Hosted job containers at once | 100 | `max_instances` of the runner container class. |
204204 | Hosted jobs at once per repository | 20 | Further jobs stay queued and start, oldest first, as the repository's own jobs finish, so one repository can't take every container. Self-hosted jobs don't count. |
205-| No free container | Wait, then retry, 8 times | When the platform has no container free for a job (many jobs starting at once, or just after a deploy), the job goes back to queued and starts again after 30 seconds, then 1, 2, 4 and 5 minutes. After 8 tries, about half an hour, it fails with `No hosted runner could be started`. Only jobs whose runner never started are retried, so no step runs twice. |
205+| No free container | Wait, then retry, 8 times | When the platform has no container free for a job (many jobs starting at once, or just after a deploy), or the job's container stops before its runner fetched the job, the job goes back to queued and starts again after 30 seconds, then 1, 2, 4 and 5 minutes. After 8 tries, about half an hour, it fails with `No hosted runner could be started`. Only jobs whose runner never started are retried, so no step runs twice. |
206206 | Unfinished runs per repository | 500 | Past it, a `workflow_dispatch` or `repository_dispatch` is refused with `429 this repository has 500 unfinished workflow runs, the most allowed; wait for some to finish`, and other events start no run. |
207207 | Self-hosted runners per repository | 100, each with at most 20 labels | Registering one more gets `429 this repository has 100 self-hosted runners, the most allowed; remove one first`. A registration token works for an hour. |
208208 | Purging unreachable objects | Repositories of up to 300,000 objects | A larger repository gets `422`; ask a site administrator. See [Removing a secret that was pushed](/docs/security/push-protection#removing-a-secret-that-was-pushed). |
modifiedapps/diffinery/src/http/gh.ts+2−0View file
@@ -651,6 +651,7 @@
651651 conclusion: string | null;
652652 startedAt: string | null;
653653 completedAt: string | null;
654+ runnerName?: string | null;
654655 steps: Array<{ index: number; name: string; status: string; conclusion?: string | null; startedAt?: string; completedAt?: string }>;
655656 };
656657
@@ -674,6 +675,7 @@
674675 workflow_name: run.workflowName,
675676 check_run_url: `${api}/check-runs/${j.id}`,
676677 labels: [] as string[],
678+ runner_name: j.runnerName ?? null,
677679 steps: j.steps.map((s) => ({ ...s, number: s.index + 1, started_at: isoSeconds(s.startedAt), completed_at: isoSeconds(s.completedAt), conclusion: s.conclusion ?? null })),
678680 };
679681 }
modifiedapps/diffinery/src/http/runner.ts+2−1View file
@@ -7,6 +7,7 @@
77 import { BodyTooLarge, putStream, sized, type BlobBucket } from "../lib/blob-store.js";
88 import { getFirst, headFirst } from "../lib/storage-keys.js";
99 import { HttpError } from "../lib/util.js";
10+import { parseRunnerInfo } from "../repo/actions.js";
1011 import { CACHE_LIMIT_BYTES } from "../repo/runtime.js";
1112 import { repoStub } from "./common.js";
1213
@@ -43,7 +44,7 @@
4344 runner.get("/runner/jobs/:jobId", async (c) => {
4445 const jobId = c.req.param("jobId");
4546 const { token, repoId } = jobAuth(c.req.raw, jobId);
46- return c.json(await repoStub(c.env, repoId).runnerSpec(jobId, token));
47+ return c.json(await repoStub(c.env, repoId).runnerSpec(jobId, token, parseRunnerInfo(c.req.header("x-diffinery-runner"))));
4748 });
4849
4950 runner.post("/runner/jobs/:jobId/logs", async (c) => {
modifiedapps/diffinery/src/repo/actions.ts+30−4View file
@@ -146,8 +146,25 @@
146146 gate_since: string | null;
147147 gate_state: string | null;
148148 placement_tries: number;
149+ runner_info: string | null;
149150 };
150151
152+/** Where a job ran, as its runner reported it: the runner image, the container's location and instance. */
153+export type JobRunnerInfo = { image?: string; location?: string; instance?: string };
154+
155+/** Reads the runner's `x-diffinery-runner` header (JSON), keeping only short string fields. */
156+export function parseRunnerInfo(header: string | null | undefined): JobRunnerInfo | null {
157+ if (!header || header.length > 1000) return null;
158+ try {
159+ const raw = JSON.parse(header) as Record<string, unknown>;
160+ const out: JobRunnerInfo = {};
161+ for (const k of ["image", "location", "instance"] as const) if (typeof raw[k] === "string" && raw[k]) out[k] = (raw[k] as string).slice(0, 200);
162+ return Object.keys(out).length ? out : null;
163+ } catch {
164+ return null;
165+ }
166+}
167+
151168 /** An environment a run's jobs wait on (GitHub's pending deployment). */
152169 export type PendingDeployment = {
153170 environment: { id: number; name: string };
@@ -198,6 +215,9 @@
198215 steps: Array<StepReport & { index: number }>;
199216 failure: JobCompletion["failure"] | null;
200217 summary: string | null;
218+ /** The runner that took the job: a self-hosted runner's name, or the hosted container's name. */
219+ runnerName: string | null;
220+ runner: JobRunnerInfo | null;
201221 };
202222
203223 const toRun = (r: RunRow): RunInfo => ({
@@ -234,6 +254,8 @@
234254 steps: JSON.parse(j.steps),
235255 failure: j.failure ? JSON.parse(j.failure) : null,
236256 summary: j.summary,
257+ runnerName: j.container?.replace(/^self-hosted:/, "") ?? null,
258+ runner: j.runner_info ? (JSON.parse(j.runner_info) as JobRunnerInfo) : null,
237259 });
238260
239261 /**
@@ -1321,8 +1343,12 @@
13211343 async containerStopped(jobId: string, exitCode: number | null): Promise<void> {
13221344 const j = this.sql.exec<JobRow>("SELECT * FROM jobs WHERE id = ?", jobId).toArray()[0];
13231345 if (!j || j.status === "completed") return;
1324- // Stopped while still waiting for its runner, without an exit code: the container never got a machine.
1325- if (j.status === "waiting" && !j.labels && !exitCode) return this.placementFailed(jobId, "the container stopped before the runner started");
1346+ // Already back in line (placementFailed): a container given up on, which started after all,
1347+ // finds its token revoked and exits. The job's next placement starts a new one.
1348+ if (j.status === "queued") return;
1349+ // Stopped before the runner fetched the job, so no step ran: place it again. During a
1350+ // container rollout a new container can exit (or never get a machine) before it reports in.
1351+ if (j.status === "waiting" && !j.labels) return this.placementFailed(jobId, exitCode ? `the runner exited with code ${exitCode} before it started the job` : "the container stopped before the runner started");
13261352 await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: exitCode ? `The runner stopped unexpectedly (exit code ${exitCode}).` : "The runner's container stopped before the job finished, without an exit code (the platform restarted or replaced it)." } });
13271353 }
13281354
@@ -1633,7 +1659,7 @@
16331659 return run.event === "pull_request" && this.repo.policy.isForkPull(run.pr_number);
16341660 }
16351661
1636- async runnerSpec(jobId: string, token: string): Promise<JobSpec> {
1662+ async runnerSpec(jobId: string, token: string, runner: JobRunnerInfo | null = null): Promise<JobSpec> {
16371663 const job = await this.authJob(jobId, token);
16381664 const run = this.runRow(job.run_id);
16391665 const exp = await this.expanded(run);
@@ -1669,7 +1695,7 @@
16691695 }
16701696 }
16711697 const t = now();
1672- this.sql.exec("UPDATE jobs SET status = 'in_progress', started_at = ? WHERE id = ?", t, jobId);
1698+ this.sql.exec("UPDATE jobs SET status = 'in_progress', started_at = ?, runner_info = ? WHERE id = ?", t, runner ? JSON.stringify(runner) : null, jobId);
16731699 this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "in_progress", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
16741700 this.repo.broadcast(`run:${run.id}`, { action: "job", jobId });
16751701 // Only the secrets the job's expressions name reach the runner, as on GitHub (plus the runner's own debug switches).
modifiedapps/diffinery/src/repo/schema.ts+2−0View file
@@ -193,6 +193,8 @@
193193 ["jobs", "runner_id", "INTEGER"],
194194 // Times a hosted job's container couldn't get a machine and the job went back in line (repo/actions.ts MAX_PLACEMENT_TRIES).
195195 ["jobs", "placement_tries", "INTEGER NOT NULL DEFAULT 0"],
196+ // What the runner reported when it fetched the job (JSON RunnerInfo: image, location, instance), for investigations.
197+ ["jobs", "runner_info", "TEXT"],
196198 // The GitHub import that wrote this row (null for everything else); a retry may only overwrite its own rows.
197199 ["issues", "imported", "TEXT"],
198200 ["pulls", "imported", "TEXT"],
modifiedapps/diffinery/test/unit/actions-runtime.test.ts+17−2View file
@@ -1,7 +1,7 @@
11 import { describe, expect, it } from "vitest";
22 import { BodyTooLarge, STEP_OPS, batchParts, blockKey, chunkKey, compose, finalKey, listStaged, parseBlockList, parseRange, planCompose, putStream, runComposeStep, stagingPrefix, type BlobBucket, type ComposeLimits, type ComposeStep } from "../../src/lib/blob-store.js";
33 import { DatabaseSync } from "node:sqlite";
4-import { Actions, MAX_ACTIVE_RUNS, MAX_HOSTED_JOBS, MAX_PLACEMENT_TRIES, placementDelay } from "../../src/repo/actions.js";
4+import { Actions, MAX_ACTIVE_RUNS, MAX_HOSTED_JOBS, MAX_PLACEMENT_TRIES, parseRunnerInfo, placementDelay } from "../../src/repo/actions.js";
55 import { testD1 } from "./stubs/d1.js";
66 import { DEFAULT_ARTIFACT_QUOTA, Runtime, artifactFileName, artifactObjectKeys } from "../../src/repo/runtime.js";
77 import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
@@ -736,6 +736,13 @@
736736 expect(job("j1")).toMatchObject({ status: "queued", placement_tries: 2 });
737737 expect(events.at(-1)).toEqual({ type: "job-j1", payload: { retryInSeconds: placementDelay(2) } });
738738 expect(running()).toEqual([]);
739+ // The same when the runner exits with a code before it fetched the job; and a container that
740+ // exits after the job went back in line (its token revoked) leaves the job in line.
741+ await actions.relaunchJob("j1");
742+ await actions.containerStopped("j1", 2);
743+ expect(job("j1")).toMatchObject({ status: "queued", conclusion: null, placement_tries: 3 });
744+ await actions.containerStopped("j1", 2);
745+ expect(job("j1")).toMatchObject({ status: "queued", conclusion: null, placement_tries: 3 });
739746 await launch("j2");
740747 sql.exec("UPDATE jobs SET status = 'in_progress' WHERE id = 'j2'");
741748 await actions.containerStopped("j2", null);
@@ -743,7 +750,7 @@
743750
744751 // Past the limit it fails, and says why.
745752 full = true;
746- for (let i = 2; i < MAX_PLACEMENT_TRIES; i++) await actions.relaunchJob("j1");
753+ for (let i = 3; i < MAX_PLACEMENT_TRIES; i++) await actions.relaunchJob("j1");
747754 expect(job("j1")).toMatchObject({ status: "queued", placement_tries: MAX_PLACEMENT_TRIES });
748755 await actions.relaunchJob("j1");
749756 expect(job("j1")).toMatchObject({ status: "completed", conclusion: "failure" });
@@ -752,6 +759,14 @@
752759 });
753760 });
754761
762+describe("runner info", () => {
763+ it("keeps only the runner's short image, location and instance strings", () => {
764+ expect(parseRunnerInfo(JSON.stringify({ image: "diffinery-runner:abc", location: "fra06, EEUR", instance: "d0", token: "x", n: 1 }))).toEqual({ image: "diffinery-runner:abc", location: "fra06, EEUR", instance: "d0" });
765+ expect(parseRunnerInfo(JSON.stringify({ image: "x".repeat(300) }))?.image).toHaveLength(200);
766+ for (const bad of [null, "", "not json", "{}", JSON.stringify({ image: 5 }), "x".repeat(2000)]) expect(parseRunnerInfo(bad)).toBeNull();
767+ });
768+});
769+
755770 describe("unfinished runs", () => {
756771 it("past MAX_ACTIVE_RUNS a dispatch is refused and other events start nothing", async () => {
757772 const sql = sqlStorage();
modifiedcontainers/runner/.dockerignore+1−0View file
@@ -1,3 +1,4 @@
11 *
22 !dist/agent.mjs
3+!dist/image-tag
34 !image/
modifiedcontainers/runner/Dockerfile+2−1View file
@@ -84,7 +84,8 @@
8484 COPY image/docker /usr/local/bin/docker
8585
8686 RUN mkdir -p /workspace /opt/hostedtoolcache /opt/diffinery
87-COPY dist/agent.mjs /opt/diffinery/agent.mjs
87+# image-tag (written by scripts/runner-image.sh) names this image in each job's log and API.
88+COPY dist/agent.mjs dist/image-tag* /opt/diffinery/
8889
8990 WORKDIR /workspace
9091 EXPOSE 8080
modifiedcontainers/runner/src/api.ts+23−2View file
@@ -1,3 +1,5 @@
1+import { readFileSync } from "node:fs";
2+import path from "node:path";
13 import type { JobCompletion, JobSpec, LogLine, StepReport } from "@diffinery/shared";
24
35 export class HttpError extends Error {
@@ -88,8 +90,9 @@
8890 throw lastErr instanceof Error ? lastErr : new Error(String(lastErr));
8991 }
9092
91- async getSpec(): Promise<JobSpec> {
92- const res = await this.request("", { retries: 8 });
93+ /** Fetches the job, telling Diffinery where it runs (`runnerInfo()`), which the job's API and log show. */
94+ async getSpec(runner: Record<string, string> = runnerInfo()): Promise<JobSpec> {
95+ const res = await this.request("", { retries: 8, headers: Object.keys(runner).length ? { "x-diffinery-runner": JSON.stringify(runner) } : {} });
9396 return (await res.json()) as JobSpec;
9497 }
9598
@@ -109,6 +112,24 @@
109112 }
110113 }
111114
115+/**
116+ * Where this runner runs: the runner image tag (written next to the agent when the hosted image is
117+ * built, see scripts/runner-image.sh) and the Cloudflare container's location and instance.
118+ */
119+export function runnerInfo(env: NodeJS.ProcessEnv = process.env, agentPath = process.argv[1]): Record<string, string> {
120+ const out: Record<string, string> = {};
121+ try {
122+ const image = agentPath ? readFileSync(path.join(path.dirname(agentPath), "image-tag"), "utf8").trim() : "";
123+ if (image) out.image = image;
124+ } catch {
125+ /* not a hosted image */
126+ }
127+ const location = [env.CLOUDFLARE_LOCATION, env.CLOUDFLARE_REGION].filter(Boolean).join(", ");
128+ if (location) out.location = location;
129+ if (env.CLOUDFLARE_DURABLE_OBJECT_ID) out.instance = env.CLOUDFLARE_DURABLE_OBJECT_ID;
130+ return out;
131+}
132+
112133 function redactUrl(u: string): string {
113134 return u.replace(/([?&](?:token|sig|signature)=)[^&]+/gi, "$1***");
114135 }
modifiedcontainers/runner/src/job.ts+3−1View file
@@ -5,7 +5,7 @@
55 import path from "node:path";
66 import type { Conclusion, ContainerSpec, JobCompletion, JobSpec, StepReport, StepSpec } from "@diffinery/shared";
77 import { evaluate, evaluateIf, interpolate, type StatusFns } from "@diffinery/workflow-yaml";
8-import type { RunnerApi } from "./api.js";
8+import { runnerInfo, type RunnerApi } from "./api.js";
99 import { parseKeyValueFile, parsePathFile } from "./commands.js";
1010 import { Docker, type Wrap } from "./docker.js";
1111 import { JobLogger, OutputProcessor, type Annotation, type OutputSink } from "./log.js";
@@ -319,6 +319,8 @@
319319 }
320320 this.defaultEnv = d;
321321 this.logger.line(-1, `Diffinery runner: job ${spec.jobKey} (${spec.name}) for ${repoFull}@${spec.sha.slice(0, 12)}`);
322+ const where = runnerInfo();
323+ if (Object.keys(where).length) this.logger.line(-1, `Runner: ${[where.image && `image ${where.image}`, where.location && `in ${where.location}`, where.instance && `container ${where.instance}`].filter(Boolean).join(", ")}`);
322324 this.logger.line(-1, `Workspace: ${this.workspace}`);
323325 // job-level env
324326 const ctx = this.contexts({ steps: {}, inputs: spec.inputs ?? {}, failed: false, inheritedEnv: {}, composite: false, depth: 0 }, {});
modifiedcontainers/runner/test/helpers.ts+2−2View file
@@ -18,7 +18,7 @@
1818 cacheVersions: Map<string, string>;
1919 /** Finalized artifacts of the run, by name. */
2020 artifacts: Map<string, { id: number; body: Buffer; mimeType: string; hash?: string }>;
21- requests: Array<{ method: string; path: string; auth: string | undefined }>;
21+ requests: Array<{ method: string; path: string; auth: string | undefined; runner?: string }>;
2222 /** Fail the next N log POSTs with 503 (to test retries). */
2323 failLogs: number;
2424 specs: Map<string, JobSpec>;
@@ -141,7 +141,7 @@
141141 for await (const c of req) chunks.push(c as Buffer);
142142 const body = Buffer.concat(chunks);
143143 const u = new URL(req.url ?? "/", "http://x");
144- api.requests.push({ method: req.method!, path: u.pathname + u.search, auth: req.headers.authorization });
144+ api.requests.push({ method: req.method!, path: u.pathname + u.search, auth: req.headers.authorization, runner: req.headers["x-diffinery-runner"] as string | undefined });
145145 const json = (status: number, v: unknown) => {
146146 res.writeHead(status, { "content-type": "application/json" });
147147 res.end(JSON.stringify(v));
modifiedcontainers/runner/test/job.test.ts+7−1View file
@@ -770,17 +770,23 @@
770770 expect(api.completion).toMatchObject({ conclusion: "success", outputs: { x: "y" } });
771771 expect(api.logText(0)).toContain("from bundle build");
772772
773- // env mode: fetch the spec from the API
773+ // env mode: fetch the spec from the API, saying where the runner runs (its image and container)
774774 api.completion = null;
775775 api.specs.set("job-2", { ...spec, jobId: "job-2", steps: [{ run: "exit 4" }] });
776+ writeFileSync(path.join(workRoot, "image-tag"), "diffinery-runner:abc123\n");
776777 const r2 = await spawnAsync(process.execPath, [out], {
777778 ...process.env,
779+ CLOUDFLARE_LOCATION: "fra06",
780+ CLOUDFLARE_REGION: "EEUR",
781+ CLOUDFLARE_DURABLE_OBJECT_ID: "d0",
778782 DIFFINERY_WORK_ROOT: path.join(workRoot, "w2"),
779783 DIFFINERY_API_URL: api.url,
780784 DIFFINERY_JOB_ID: "job-2",
781785 DIFFINERY_JOB_TOKEN: api.token,
782786 });
783787 expect(r2.status).toBe(1);
788+ expect(JSON.parse(api.requests.find((q) => q.path === "/runner/jobs/job-2")?.runner ?? "{}")).toEqual({ image: "diffinery-runner:abc123", location: "fra06, EEUR", instance: "d0" });
789+ expect(api.logText()).toContain("Runner: image diffinery-runner:abc123, in fra06, EEUR, container d0");
784790 expect(api.completion).toMatchObject({ conclusion: "failure", failure: { step: 0 } });
785791 expect(readFileSync(path.join(workRoot, "w2/_temp/_github_workflow/event.json"), "utf8")).toContain("pusher");
786792
modifiedscripts/runner-image.sh+2−0View file
@@ -31,6 +31,8 @@
3131 echo "reusing $name:$tag: the runner hasn't changed since it was built"
3232 else
3333 echo "building $name:$tag"
34+ # The runner reports this as its image (not part of the tag's hash, which lists its inputs).
35+ echo "$name:$tag" > "$ctx/dist/image-tag"
3436 wrangler containers build "$PWD/$ctx" -t "$name:$tag" -p
3537 fi
3638 [ -z "${GITHUB_ENV:-}" ] || echo "RUNNER_IMAGE=$image" >> "$GITHUB_ENV"
modifiedscripts/wait-container-rollout.sh+2−1View file
@@ -25,7 +25,8 @@
2525 r=$(api "/applications/$app/rollouts/$rollout" | jq -c .result) || continue
2626 # Only the rollouts of this deploy's image (matched by its tag, a hash of the runner).
2727 [[ "$(jq -r '.target_configuration.image // ""' <<<"$r")" == *":${RUNNER_IMAGE##*:}" ]] || continue
28- state=$(jq -r '"\(.status), step \((.progress.current_step // 0) + 1) of \(.progress.total_steps // (.steps | length)) \((.steps // []) | last | .status // ""), \(.progress.updated_instances // "?")/\(.progress.total_instances // "?") instances updated"' <<<"$r")
28+ # Steps started out of all steps, from the steps themselves (progress.current_step is 1-based in practice).
29+ state=$(jq -r '(.steps // []) as $s | "\(.status), step \([$s[] | select(.status != "pending")] | length) of \($s | length) \($s | last | .status // ""), \(.progress.updated_instances // "?")/\(.progress.total_instances // "?") instances updated"' <<<"$r")
2930 states+="$name: $state; "
3031 # Done once the rollout ended, or its last step is under way (new containers get the new image).
3132 jq -e '(.status | IN("completed", "reverted", "replaced")) or ((.steps // []) | length > 0 and (last | .status | IN("progressing", "completed")))' <<<"$r" >/dev/null || pending+="$name "