Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

End a running job whose runner stopped responding 10 minutes after its timeout (#227)

* End a running job whose runner stopped responding 10 minutes after its timeout
test-bot authored 1 parent b7afa38commit 7ceb5139d7846eabcc61cd294b6adeecfd234b2fBrowse files
5 files changed+139−8
Files (5)
modifiedapps/diffinery/docs/actions/runners.md+1−0View file
@@ -180,6 +180,7 @@
180180 | Job timeout | `timeout-minutes`, default 360 minutes, and values above 360 are capped at 360 | The job fails with "The job running on runner has exceeded the maximum execution time of N minutes." |
181181 | Total wait per job | 6 hours from when the job is scheduled, including time spent waiting for a runner | The job fails with "The job exceeded its timeout of N minutes." |
182182 | Step timeout | `steps[*].timeout-minutes`, no default | The step fails with "The step has timed out after N minutes." |
183+| Runner that stops responding | 10 minutes past the job timeout | The runner ends a job at its timeout itself and reports within seconds. If it hasn't reported 10 minutes later (its agent froze or its machine vanished), Diffinery stops the container and the job fails as timed out with "The job exceeded its timeout of N minutes, and its runner stopped responding: Diffinery ended it 10 minutes later." So a stuck runner holds a hosted container, a slot of the account's hosted jobs and its minutes for at most the job's timeout plus 10 minutes. Set `timeout-minutes` on jobs that should never run long. |
183184 | Matrix size | 256 jobs per matrix | The job fails with "A matrix may produce at most 256 jobs." |
184185 | Log line length | 16,384 characters | Longer lines are cut and end with `…` |
185186 | Log size | 16 MiB of text per job | The log ends with "The log is cut off here: a job's log is limited to 16 MiB of text." Later lines are dropped; the job keeps running. |
modifiedapps/diffinery/docs/reference/limits.md+1−0View file
@@ -211,6 +211,7 @@
211211 | `workflow_run` chains | 3 levels | A run started by `workflow_run` that is itself three links down a chain triggers nothing (A starts B starts C starts D; D's runs start nothing). |
212212 | Schedule granularity | 5 minutes | Scheduled workflows are checked every 5 minutes. A cron that fires more often still runs at most once per 5-minute window. Only workflows on the default branch of repositories that aren't archived are scheduled. |
213213 | Schedule catch-up | 60 minutes | A check that comes late, or fails, is caught up by the next one, up to an hour back. Each schedule starts once per check, however many of its minutes the check covers. |
214+| Running job whose runner stops responding | Ended 10 minutes after its `timeout-minutes` | Counted from when its runner fetched the job. The runner ends a job at its timeout itself; Diffinery only steps in when it hasn't reported by then, stops its container and fails it as timed out. |
214215 | Starting a job | 3 retries | Starting a job's container is retried 3 times with exponential backoff from 10 seconds before the job fails with "Could not start a runner". |
215216 | Hosted runner sizes | large: `standard-4` (4 vCPU, 12 GiB of memory, 20 GB of disk); medium: `standard-3` (2 vCPU, 8 GiB, 16 GB); small: `standard-1` (1/2 vCPU, 4 GiB, 8 GB); tiny: `basic` (1/4 vCPU, 1 GiB, 4 GB) | Each hosted job runs in its own container. A job runs on large unless `runs-on` names a size or its earlier runs show a smaller one fits with headroom ([Runner sizes](/docs/actions/runners#runner-sizes)). |
216217 | Hosted job containers at once | 100 per size | `max_instances` of each runner container class. |
modifiedapps/diffinery/src/repo/RepoDO.ts+24−7View file
@@ -320,23 +320,40 @@
320320
321321 /**
322322 * Set (or with null, clear) the object's single alarm. The merge queue uses it for check timeouts;
323- * while jobs wait on the account's hosted-job quota it also fires every minute to retry them.
323+ * while jobs wait on the account's hosted-job quota it also fires every minute to retry them, and it
324+ * fires at the earliest running job's deadline (Actions.enforceJobDeadlines).
324325 */
325326 armAlarm(at: number | null): void {
326- const held = this.actions.hasAccountHeldJobs() ? Date.now() + HELD_RETRY_MS : null;
327- const next = at === null ? held : held === null ? at : Math.min(at, held);
327+ const next = this.earliest(at, this.actions.hasAccountHeldJobs() ? Date.now() + HELD_RETRY_MS : null, this.actions.nextJobDeadline());
328328 this.ctx.waitUntil(next === null ? this.ctx.storage.deleteAlarm() : this.ctx.storage.setAlarm(next));
329329 }
330330
331+ /** Make sure the alarm fires by `at`, keeping an earlier one. */
332+ alarmBy(at: number): void {
333+ this.ctx.waitUntil(
334+ (async () => {
335+ const current = await this.ctx.storage.getAlarm();
336+ if (current === null || current > at) await this.ctx.storage.setAlarm(at);
337+ })(),
338+ );
339+ }
340+
341+ private earliest(...times: Array<number | null>): number | null {
342+ const set = times.filter((t): t is number => t !== null);
343+ return set.length ? Math.min(...set) : null;
344+ }
345+
331346 override async alarm(): Promise<void> {
332347 const purging = await this.ctx.storage.get<string>(PURGE_KEY);
333348 if (purging) return this.purge(purging);
334349 await this.queue.process();
335- if (this.actions.hasAccountHeldJobs()) {
336- await this.actions.releaseAccountHeld();
337- // Keep the queue's own alarm if it set an earlier one.
350+ if (this.actions.hasAccountHeldJobs()) await this.actions.releaseAccountHeld();
351+ await this.actions.enforceJobDeadlines();
352+ // Wake up again for held jobs and running jobs' deadlines, keeping the queue's own alarm if it set an earlier one.
353+ const want = this.earliest(this.actions.hasAccountHeldJobs() ? Date.now() + HELD_RETRY_MS : null, this.actions.nextJobDeadline());
354+ if (want !== null) {
338355 const current = await this.ctx.storage.getAlarm();
339- if (this.actions.hasAccountHeldJobs() && (current === null || current <= Date.now() || current > Date.now() + HELD_RETRY_MS)) await this.ctx.storage.setAlarm(Date.now() + HELD_RETRY_MS);
356+ if (current === null || current <= Date.now() || current > want) await this.ctx.storage.setAlarm(want);
340357 }
341358 }
342359
modifiedapps/diffinery/src/repo/actions.ts+47−0View file
@@ -43,6 +43,15 @@
4343 export const SANDBOX_TOKEN_TTL_MS = 24 * 3600_000;
4444 /** A job token works this long past the job's timeout, so a runner can still report a job that hit it. */
4545 const JOB_TOKEN_MARGIN_MS = 30 * 60_000;
46+/**
47+ * How long past its timeout-minutes (from when its runner fetched it) a running job may go without
48+ * reporting before Diffinery ends it itself. The runner stops a job at its timeout and reports within
49+ * seconds (it cancels the steps and skips post steps), so this only ends jobs whose runner agent froze
50+ * or vanished, which would otherwise hold their container, quota slot and minutes for up to 6 hours.
51+ */
52+export const JOB_DEADLINE_GRACE_MS = 10 * 60_000;
53+/** timeout-minutes when a job row has none (GitHub's default). */
54+const DEFAULT_TIMEOUT_MINUTES = 360;
4655 /** A job token never works longer than this after the job started (timeouts are at most 360 minutes). */
4756 export const MAX_JOB_TOKEN_TTL_MS = 7 * 3600_000;
4857
@@ -1662,6 +1671,42 @@
16621671 await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: `The job exceeded its timeout of ${j.timeout_minutes} minutes.` } }, "timed_out");
16631672 }
16641673
1674+ /** When a running job's runner has had its timeout and JOB_DEADLINE_GRACE_MS to report, in ms since the epoch. */
1675+ private jobDeadline(j: Pick<JobRow, "started_at" | "timeout_minutes">): number | null {
1676+ const from = Date.parse(j.started_at ?? "");
1677+ if (!Number.isFinite(from)) return null;
1678+ const minutes = Number(j.timeout_minutes) > 0 ? Number(j.timeout_minutes) : DEFAULT_TIMEOUT_MINUTES;
1679+ return from + minutes * 60_000 + JOB_DEADLINE_GRACE_MS;
1680+ }
1681+
1682+ /** The earliest deadline of a running job (RepoDO's alarm wakes up for it), or null. */
1683+ nextJobDeadline(): number | null {
1684+ let next: number | null = null;
1685+ for (const j of this.sql.exec<JobRow>("SELECT started_at, timeout_minutes FROM jobs WHERE status = 'in_progress' AND started_at IS NOT NULL").toArray()) {
1686+ const d = this.jobDeadline(j);
1687+ if (d !== null && (next === null || d < next)) next = d;
1688+ }
1689+ return next;
1690+ }
1691+
1692+ /**
1693+ * Ends running jobs past their deadline: their runner never reported, though it stops a job at its
1694+ * timeout. Its container is stopped and the job fails as timed out, freeing the slot it held.
1695+ */
1696+ async enforceJobDeadlines(at = Date.now()): Promise<void> {
1697+ for (const j of this.sql.exec<JobRow>("SELECT * FROM jobs WHERE status = 'in_progress' AND started_at IS NOT NULL ORDER BY rowid").toArray()) {
1698+ const d = this.jobDeadline(j);
1699+ if (d === null || d > at) continue;
1700+ const minutes = Number(j.timeout_minutes) > 0 ? Number(j.timeout_minutes) : DEFAULT_TIMEOUT_MINUTES;
1701+ await this.stopContainer(j);
1702+ await this.completeJob(
1703+ j.id,
1704+ { conclusion: "failure", outputs: {}, failure: { message: `The job exceeded its timeout of ${minutes} minutes, and its runner stopped responding: Diffinery ended it ${JOB_DEADLINE_GRACE_MS / 60_000} minutes later.` } },
1705+ "timed_out",
1706+ );
1707+ }
1708+ }
1709+
16651710 async containerStopped(jobId: string, exitCode: number | null, container?: string, requested = false): Promise<void> {
16661711 const j = this.sql.exec<JobRow>("SELECT * FROM jobs WHERE id = ?", jobId).toArray()[0];
16671712 if (!j || j.status === "completed") return;
@@ -2120,6 +2165,8 @@
21202165 }
21212166 const t = now();
21222167 this.sql.exec("UPDATE jobs SET status = 'in_progress', started_at = ?, runner_info = ? WHERE id = ?", t, runner ? JSON.stringify(runner) : null, jobId);
2168+ // Wake up at its deadline in case its runner freezes (enforceJobDeadlines).
2169+ this.repo.alarmBy(this.jobDeadline({ started_at: t, timeout_minutes: job.timeout_minutes })!);
21232170 this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "in_progress", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
21242171 this.repo.broadcast(`run:${run.id}`, { action: "job", jobId });
21252172 // Only the secrets the job's expressions name reach the runner, as on GitHub (plus the runner's own debug switches).
modifiedapps/diffinery/test/unit/actions-runtime.test.ts+66−1View file
@@ -1,7 +1,7 @@
11 import { describe, expect, it } from "vitest";
22 import { BodyTooLarge, STEP_OPS, batchParts, blockKey, chunkKey, compose, finalKey, listStaged, parseBlockList, parseRange, planCompose, putStream, runComposeStep, stagingPrefix, type BlobBucket, type ComposeLimits, type ComposeStep } from "../../src/lib/blob-store.js";
33 import { DatabaseSync } from "node:sqlite";
4-import { Actions, MAX_ACTIVE_RUNS, MAX_PLACEMENT_TRIES, MAX_RUNNER_RESTARTS, parseRunnerInfo, placementDelay, QUICK_RETRY_SECONDS, STALE_IMAGE_TRIES } from "../../src/repo/actions.js";
4+import { Actions, JOB_DEADLINE_GRACE_MS, MAX_ACTIVE_RUNS, MAX_PLACEMENT_TRIES, MAX_RUNNER_RESTARTS, parseRunnerInfo, placementDelay, QUICK_RETRY_SECONDS, STALE_IMAGE_TRIES } from "../../src/repo/actions.js";
55 import { fleetLimits } from "../../src/lib/quotas.js";
66 import { testD1 } from "./stubs/d1.js";
77 import { DEFAULT_ARTIFACT_QUOTA, Runtime, artifactFileName, artifactObjectKeys } from "../../src/repo/runtime.js";
@@ -926,6 +926,71 @@
926926 await expect(refuse("s1", "my-image")).resolves.toBeUndefined();
927927 });
928928
929+ it("a running job whose runner froze is ended at its timeout plus JOB_DEADLINE_GRACE_MS, and a healthy one is left alone", async () => {
930+ const sql = sqlStorage();
931+ sql.exec(REPO_SCHEMA);
932+ migrateRepoSchema(sql as unknown as SqlStorage);
933+ const events: { type: string; payload: Record<string, unknown> }[] = [];
934+ const stopped: string[] = [];
935+ const repo = {
936+ sql,
937+ meta: { repoId: "r", owner: "acme", ownerId: "acme", name: "app", defaultBranch: "main" },
938+ upsertCheck: () => {},
939+ broadcast: () => {},
940+ deploys: { jobCompleted: () => {} },
941+ runtime: { sweepJob: async () => {} },
942+ } as unknown as RepoDO;
943+ const env = {
944+ PUBLIC_URL: "https://d.test",
945+ BUCKET: memoryBucket().bucket,
946+ RUNNER: { idFromName: (n: string) => n, get: (n: string) => ({ startJob: async () => {}, stopJob: async () => void stopped.push(n) }) },
947+ RUN_WORKFLOW: { get: async () => ({ sendEvent: async (e: { type: string; payload: Record<string, unknown> }) => void events.push(e) }) },
948+ } as unknown as Env;
949+ const actions = new Actions(repo, env);
950+ const t0 = Date.parse("2026-10-16T12:00:00Z");
951+ const at = (ms: number) => new Date(t0 + ms).toISOString();
952+ sql.exec("INSERT INTO runs (id, run_number, workflow_path, workflow_name, event, ref, sha, actor, status, created_at, updated_at, event_payload) VALUES ('run1', 1, 'w.yml', 'w', 'push', 'refs/heads/main', 'abc', '{}', 'in_progress', ?, ?, '{}')", at(0), at(0));
953+ // Each runner fetched its job at t0 (started_at). f: frozen, timeout 10 min; h: healthy, timeout 30;
954+ // d: no timeout of its own (6 hours); s: on a self-hosted runner, frozen too; q: still in line; c: finished.
955+ const add = (id: string, minutes: number | null, o: { status?: string; container?: string | null; started?: string | null } = {}) =>
956+ sql.exec(
957+ "INSERT INTO jobs (id, run_id, job_key, name, status, timeout_minutes, container, started_at, strategy) VALUES (?, 'run1', ?, ?, ?, ?, ?, ?, '{\"fail-fast\":false}')",
958+ id, id, id, o.status ?? "in_progress", minutes, o.container === undefined ? `job-${id}` : o.container, o.started === undefined ? at(0) : o.started,
959+ );
960+ add("f", 10);
961+ add("h", 30);
962+ add("d", 0);
963+ add("s", 10, { container: "self-hosted:box" });
964+ add("q", 10, { status: "queued", container: null, started: null });
965+ add("c", 10, { status: "completed" });
966+ const job = (id: string) => sql.exec<{ status: string; conclusion: string | null; failure: string | null }>("SELECT status, conclusion, failure FROM jobs WHERE id = ?", id).one();
967+ const min = 60_000;
968+ expect(actions.nextJobDeadline()).toBe(t0 + 10 * min + JOB_DEADLINE_GRACE_MS);
969+
970+ // Past f's timeout but within the grace its runner has to stop the steps and report: nothing happens.
971+ await actions.enforceJobDeadlines(t0 + 10 * min + JOB_DEADLINE_GRACE_MS - 1);
972+ expect(["f", "h", "d", "s"].map((id) => job(id).status)).toEqual(["in_progress", "in_progress", "in_progress", "in_progress"]);
973+ expect(stopped).toEqual([]);
974+
975+ // At the deadline the frozen ones end: the hosted container is stopped, both fail as timed out.
976+ await actions.enforceJobDeadlines(t0 + 10 * min + JOB_DEADLINE_GRACE_MS);
977+ for (const id of ["f", "s"]) {
978+ expect(job(id)).toMatchObject({ status: "completed", conclusion: "timed_out" });
979+ expect(JSON.parse(job(id).failure!).message).toBe(`The job exceeded its timeout of 10 minutes, and its runner stopped responding: Diffinery ended it ${JOB_DEADLINE_GRACE_MS / min} minutes later.`);
980+ }
981+ expect(stopped).toEqual(["job-f"]);
982+ expect(events.filter((e) => e.type === "job-f")).toEqual([{ type: "job-f", payload: { conclusion: "timed_out" } }]);
983+ // The others are untouched, and the next wake-up is h's deadline.
984+ expect([job("h").status, job("d").status, job("q").status, job("c").status]).toEqual(["in_progress", "in_progress", "queued", "completed"]);
985+ expect(actions.nextJobDeadline()).toBe(t0 + 30 * min + JOB_DEADLINE_GRACE_MS);
986+ // A job without timeout-minutes gets GitHub's 6 hours.
987+ await actions.enforceJobDeadlines(t0 + 360 * min + JOB_DEADLINE_GRACE_MS - 1);
988+ expect(job("d").status).toBe("in_progress");
989+ await actions.enforceJobDeadlines(t0 + 360 * min + JOB_DEADLINE_GRACE_MS);
990+ expect(job("d")).toMatchObject({ status: "completed", conclusion: "timed_out" });
991+ expect(actions.nextJobDeadline()).toBeNull();
992+ });
993+
929994 it("a job whose runner a rollout replaced starts again on a new runner before its steps ran or when it can only read, and fails otherwise", async () => {
930995 const sql = sqlStorage();
931996 sql.exec(REPO_SCHEMA);