Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Runners: 4 vCPU, matrix legs start together, and a slow container start is waited for (#186)

* Runners: 4 vCPU, matrix legs start together, and a slow container start is waited for
* A repository whose default branch wasn't pushed shows its branches instead of a 404
test-bot authored 1 parent 844812fcommit effb7caf7edead436a659ba45594e585320b1e70Browse files
9 files changed+282−33
Files (9)
modifiedapps/diffinery/docs/actions/runners.md+1−1View file
@@ -62,7 +62,7 @@
6262
6363 | Property | Value |
6464 |---|---|
65-| Platform | Cloudflare Containers, instance type `standard-2` (see Cloudflare's Containers documentation for its vCPU, memory and disk) |
65+| Platform | Cloudflare Containers, instance type `standard-4`: 4 vCPU, 12 GiB of memory, 20 GB of disk |
6666 | Architecture | `linux/amd64` (`runner.arch` is `X64`) |
6767 | Operating system | Ubuntu 24.04 (`runner.os` is `Linux`) |
6868 | User | `root`. `sudo` is installed and needs no password, so `sudo apt-get install ...` works as on GitHub |
modifiedapps/diffinery/docs/reference/limits.md+2−2View file
@@ -211,7 +211,7 @@
211211 | Schedule granularity | 5 minutes | Scheduled workflows are checked every 5 minutes. A cron that fires more often still runs at most once per 5-minute window. Only workflows on the default branch of repositories that aren't archived are scheduled. |
212212 | Schedule catch-up | 60 minutes | A check that comes late, or fails, is caught up by the next one, up to an hour back. Each schedule starts once per check, however many of its minutes the check covers. |
213213 | Starting a job | 3 retries | Starting a job's container is retried 3 times with exponential backoff from 10 seconds before the job fails with "Could not start a runner". |
214-| Hosted runner size | Cloudflare Containers `standard-2` | Each `runs-on: ubuntu-latest` job runs in its own container of this instance type. |
214+| Hosted runner size | Cloudflare Containers `standard-4`: 4 vCPU, 12 GiB of memory, 20 GB of disk | Each `runs-on: ubuntu-latest` job runs in its own container of this instance type. |
215215 | Hosted job containers at once | 100 | `max_instances` of the runner container class. |
216216 | Hosted jobs at once per repository | 20 | Further jobs stay queued and start, oldest first, as the repository's own jobs finish, so one repository can't take every container. Self-hosted jobs don't count. |
217217 | No free container | Wait, then retry, 8 times | When the platform has no container free for a job (many jobs starting at once, or just after a deploy), the start fails on the platform's side in a way that passes ("Container sidecar is shutting down", a dropped connection), or the job's container stops before its runner fetched the job, the job goes back to queued and starts again after 30 seconds, then 1, 2, 4 and 5 minutes. After 8 tries, about half an hour, it fails with `No hosted runner could be started`. Only jobs whose runner never started are retried, so no step runs twice. |
@@ -378,7 +378,7 @@
378378 | --- | --- | --- |
379379 | Worker CPU time | 300,000 ms per request | `limits.cpu_ms`. |
380380 | Live update connection | 10 minutes, up to 10 topics | A page's WebSocket is closed with code `4001` after 10 minutes and reconnects at once, so access is checked again. |
381-| Runner containers | `standard-2`, up to 100 instances | `containers[RunnerContainer]`. |
381+| Runner containers | `standard-4`, up to 100 instances | `containers[RunnerContainer]`. |
382382 | Sandbox containers | `standard-1`, up to 50 instances | `containers[SandboxContainer]`. |
383383 | Import containers | `standard-4`, up to 20 instances | `containers[ImportContainer]`: one container per Git transfer of an import. |
384384 | Container rollout grace period | 23,400 seconds for runners, 25,200 for imports, 3,600 for sandboxes | `rollout_active_grace_period`. A deploy doesn't replace a container younger than this, so running jobs (6 hours at most) and import transfers finish first. |
modifiedapps/diffinery/docs/self-hosting/architecture.md+1−1View file
@@ -170,7 +170,7 @@
170170
171171 All three container classes use the same image, built from `containers/runner/Dockerfile` (Ubuntu 24.04 with Node.js 22, git, git-lfs, Python and build tools, plus the runner agent).
172172
173-- **`RunnerContainer`** runs one hosted job (`runs-on: ubuntu-latest` and other `ubuntu-*` or `linux` labels). The Durable Object starts a container named after the job with three environment variables: `DIFFINERY_API_URL`, `DIFFINERY_JOB_ID` and `DIFFINERY_JOB_TOKEN`. The agent fetches its job spec from `/runner/jobs/<id>`, runs the steps, streams logs, uses the cache and artifact services and the OIDC endpoint, reports completion and exits. If the container stops before the job reports completion, the job fails with "The runner stopped unexpectedly". Instance type `standard-2`, at most 100 at once, internet access on (only HTTP and HTTPS, through the Worker, under [`EGRESS=web`](/docs/self-hosting/configuration#egress)). Docker is started inside the container the first time a job uses it.
173+- **`RunnerContainer`** runs one hosted job (`runs-on: ubuntu-latest` and other `ubuntu-*` or `linux` labels). The Durable Object starts a container named after the job with three environment variables: `DIFFINERY_API_URL`, `DIFFINERY_JOB_ID` and `DIFFINERY_JOB_TOKEN`. The agent fetches its job spec from `/runner/jobs/<id>`, runs the steps, streams logs, uses the cache and artifact services and the OIDC endpoint, reports completion and exits. If the container stops before the job reports completion, the job fails with "The runner stopped unexpectedly". Instance type `standard-4` (4 vCPU, 12 GiB), at most 100 at once, internet access on (only HTTP and HTTPS, through the Worker, under [`EGRESS=web`](/docs/self-hosting/configuration#egress)). Docker is started inside the container the first time a job uses it.
174174 - **`SandboxContainer`** runs one agent sandbox: the repository checked out at a ref, with an HTTP API on port 8080 for `exec`, file reads and file writes. The Worker forwards REST and MCP sandbox calls to it with a per-sandbox secret. Instance type `standard-1`, at most 50 at once, stopped after 30 minutes without requests. Internet access is on, limited to HTTP and HTTPS under `EGRESS=web` like `RunnerContainer`, and a limited sandbox stays limited when it starts again after its idle stop.
175175 - **`ImportContainer`** runs the Git transfer of one GitHub import: the runner agent in import mode, started with `DIFFINERY_API_URL`, `DIFFINERY_IMPORT_ID` and `DIFFINERY_IMPORT_TOKEN`. One container per try, named `<import id>-<attempt>-<try>`. If it stops before the agent reports a result (a crash, running out of memory, a platform restart), the transfer counts as failed and the import's Workflow can try it again. Instance type `standard-4` (a mirror clone of a large repository needs disk and memory), at most 20 at once, internet access on, stopped after 7 hours without activity.
176176
modifiedapps/diffinery/docs/self-hosting/configuration.md+1−1View file
@@ -210,7 +210,7 @@
210210 | `observability` | Enabled, `head_sampling_rate: 1` | Every request is logged to Workers Logs. |
211211 | `upload_source_maps` | `true` | Readable stack traces in the logs. |
212212 | `limits.cpu_ms` | `300000` | Up to 5 minutes of CPU time per request (large pushes, diffs and merges). |
213-| `containers` | `RunnerContainer`: `standard-2`, `max_instances: 100`; `SandboxContainer`: `standard-1`, `max_instances: 50`; `ImportContainer`: `standard-4`, `max_instances: 20`; `rollout_active_grace_period`: 23400 for runners and 25200 for imports (longer than a job or transfer can run, so a deploy never replaces a running one), 3600 for sandboxes; `rollout_step_percentage: 100`, so containers started after a deploy get the new image | All three build `containers/runner/Dockerfile` with `containers/runner` as context. Raise `max_instances` for more concurrent jobs, sandboxes or import transfers. |
213+| `containers` | `RunnerContainer`: `standard-4`, `max_instances: 100`; `SandboxContainer`: `standard-1`, `max_instances: 50`; `ImportContainer`: `standard-4`, `max_instances: 20`; `rollout_active_grace_period`: 23400 for runners and 25200 for imports (longer than a job or transfer can run, so a deploy never replaces a running one), 3600 for sandboxes; `rollout_step_percentage: 100`, so containers started after a deploy get the new image | All three build `containers/runner/Dockerfile` with `containers/runner` as context. Raise `max_instances` for more concurrent jobs, sandboxes or import transfers. |
214214 | `triggers.crons` | `["*/5 * * * *"]` | Token expiry reminders, email log pruning, registry cleanup, cancelling draft imports older than 7 days, and setting the workflow scheduler's alarm if it isn't set. Scheduled workflows run from that alarm, not from the cron trigger. |
215215
216216 ## The dev environment
modifiedapps/diffinery/src/repo/actions.ts+73−20View file
@@ -23,7 +23,7 @@
2323 import { ghRepo, numericId } from "../http/gh.js";
2424 import { HttpError, newId, now, randomToken, sha256 } from "../lib/util.js";
2525 import { localOnly } from "../auth/dev.js";
26-import { egressFor } from "../lib/egress.js";
26+import { egressFor, type Egress } from "../lib/egress.js";
2727 import { endCompute, mayStart, startCompute } from "../lib/quotas.js";
2828 import { blockedBy, killSwitches, type Work } from "../lib/killswitch.js";
2929 import type { Env } from "../env.js";
@@ -76,9 +76,11 @@
7676 /**
7777 * A container start that failed on the platform's side and works when tried again: no machine free,
7878 * a container sidecar or Durable Object restarting under it (seen during rollouts as "Container
79- * sidecar is shutting down"), or a dropped connection to it.
79+ * sidecar is shutting down"), a dropped connection to it, or a container that didn't come up in time.
8080 */
81-const RETRYABLE_START = /no container instance|is shutting down|network connection lost|durable object reset|overloaded/i;
81+const RETRYABLE_START = /no container instance|is shutting down|network connection lost|durable object reset|overloaded|too many containers|aborted waiting for container to start/i;
82+/** Hosted containers one matrix starts at once: each start takes a couple of seconds. */
83+const START_AT_ONCE = 8;
8284 /** Runs one repository may have unfinished at once; past it, new ones aren't started. */
8385 export const MAX_ACTIVE_RUNS = 500;
8486 /** Self-hosted runners one repository may have registered. */
@@ -88,6 +90,9 @@
8890 /** A runner that polled this recently shows as online. */
8991 const RUNNER_ONLINE_MS = 90_000;
9092
93+/** A hosted job admitted to start (Actions.admit), and what its container starts with. */
94+type Placement = { jobId: string; container: string; token: string; egress: Egress };
95+
9196 type RunnerRow = { id: number; name: string; labels: string; token_hash: string | null; ephemeral: number; created_by: string; created_at: string; last_seen_at: string | null };
9297
9398 export interface RunnerInfo {
@@ -1074,7 +1079,7 @@
10741079 );
10751080 ids.push(id);
10761081 });
1077- for (const id of ids.slice(0, maxParallel)) await this.launch(id);
1082+ await this.launchAll(ids.slice(0, maxParallel));
10781083 this.repo.broadcast(`run:${runId}`, { action: "jobs" });
10791084 return { jobIds: ids };
10801085 }
@@ -1117,19 +1122,65 @@
11171122 }
11181123
11191124 private async launch(jobId: string): Promise<void> {
1120- const job = this.jobRow(jobId);
1125+ const placement = await this.admit(jobId);
1126+ if (placement) await this.place(placement);
1127+ }
1128+
1129+ /**
1130+ * Starts jobs (matrix legs) together. Each is admitted (kill switches, gates, quotas, the
1131+ * repository's share of the fleet) one after another, so no two take the same slot; their
1132+ * containers then start START_AT_ONCE at a time, since one start takes seconds. Once a leg has to
1133+ * wait for a slot, so would the rest: they wait in line with it without asking again.
1134+ */
1135+ private async launchAll(jobIds: string[]): Promise<void> {
1136+ const starting = new Set<Promise<void>>();
1137+ let heldAs = 0;
1138+ for (const id of jobIds) {
1139+ if (heldAs) {
1140+ this.holdForSlot(this.jobRow(id), heldAs);
1141+ continue;
1142+ }
1143+ const placement = await this.admit(id);
1144+ if (!placement) {
1145+ const j = this.jobRow(id);
1146+ if (j.status === "queued" && j.fleet_held > 0) heldAs = j.fleet_held;
1147+ continue;
1148+ }
1149+ const started: Promise<void> = this.place(placement).finally(() => starting.delete(started));
1150+ starting.add(started);
1151+ if (starting.size >= START_AT_ONCE) await Promise.race(starting);
1152+ }
1153+ await Promise.all(starting);
1154+ }
1155+
1156+ /** Puts a job in line for a hosted slot: 1 the repository's share of the fleet, 2 the account's hosted jobs. */
1157+ private holdForSlot(job: JobRow, held: number): void {
11211158 if (job.status === "completed") return;
1159+ const run = this.runRow(job.run_id);
1160+ const m = this.repo.meta;
1161+ this.sql.exec("UPDATE jobs SET status = 'queued', fleet_held = ? WHERE id = ?", held, job.id);
1162+ this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId: job.id, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${job.id}` });
1163+ if (held === 2) this.repo.armAlarm(null);
1164+ }
1165+
1166+ /** Everything before a hosted job's container starts; null when it doesn't start now (finished, parked or held). */
1167+ private async admit(jobId: string): Promise<Placement | null> {
1168+ const job = this.jobRow(jobId);
1169+ if (job.status === "completed") return null;
11221170 // Kill switches (lib/killswitch.ts): a suspended repository or owner starts no job; a paused instance no hosted one.
11231171 const block = await this.blocked(job.labels ? "self-hosted-job" : "hosted-job");
1124- if (block) return this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: `The job was not started: ${block}.` } });
1125- if (!(await this.gate(job)) || !(await this.concurrencySlot(job))) return;
1172+ if (block) {
1173+ await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: `The job was not started: ${block}.` } });
1174+ return null;
1175+ }
1176+ if (!(await this.gate(job)) || !(await this.concurrencySlot(job))) return null;
11261177 const run = this.runRow(job.run_id);
11271178 const m = this.repo.meta;
11281179 if (job.labels) {
11291180 // Self-hosted: park the job until a registered runner with matching labels claims it.
11301181 this.sql.exec("UPDATE jobs SET status = 'waiting', container = NULL WHERE id = ?", jobId);
11311182 this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
1132- return;
1183+ return null;
11331184 }
11341185 // The account that owns the repository pays (lib/quotas.ts): past its monthly minutes the job
11351186 // fails; past its hosted jobs at once it waits (fleet_held = 2, retried by the alarm) for one to end.
@@ -1138,18 +1189,15 @@
11381189 return { ok: true } as const;
11391190 });
11401191 if (!quota.ok) {
1141- if (!quota.wait) return this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: `The job was not started: ${quota.message}.` } });
1142- this.sql.exec("UPDATE jobs SET status = 'queued', fleet_held = 2 WHERE id = ?", jobId);
1143- this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
1144- this.repo.armAlarm(null);
1145- return;
1192+ if (quota.wait) this.holdForSlot(job, 2);
1193+ else await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: `The job was not started: ${quota.message}.` } });
1194+ return null;
11461195 }
11471196 // One repository can't take the whole fleet: past its share, the job waits for one of its own to finish.
11481197 const busy = this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM jobs WHERE labels IS NULL AND container IS NOT NULL AND status IN ('waiting', 'in_progress') AND id != ?", jobId).one().n;
11491198 if (busy >= MAX_HOSTED_JOBS) {
1150- this.sql.exec("UPDATE jobs SET status = 'queued', fleet_held = 1 WHERE id = ?", jobId);
1151- this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
1152- return;
1199+ this.holdForSlot(job, 1);
1200+ return null;
11531201 }
11541202 this.sql.exec("UPDATE jobs SET fleet_held = 0 WHERE id = ?", jobId);
11551203 const token = `djt_${m.repoId}_${jobId}_${randomToken(24)}`;
@@ -1158,13 +1206,18 @@
11581206 this.sql.exec("UPDATE jobs SET status = 'waiting', token_hash = ?, container = ? WHERE id = ?", await sha256(token), container, jobId);
11591207 await startCompute(this.env, { id: jobId, accountId: m.ownerId, repoId: m.repoId, kind: "job" }).catch((e: unknown) => console.error("compute usage failed", e));
11601208 this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
1209+ return { jobId, container, token, egress: await egressFor(this.env, m.ownerId) };
1210+ }
1211+
1212+ /** Starts an admitted job's container. One that got no machine goes back in line (placementFailed). */
1213+ private async place(p: Placement): Promise<void> {
11611214 try {
1162- const stub = this.env.RUNNER.get(this.env.RUNNER.idFromName(container));
1163- await stub.startJob({ repoId: m.repoId, jobId, container, token, apiUrl: this.env.PUBLIC_URL, egress: await egressFor(this.env, m.ownerId) });
1215+ const stub = this.env.RUNNER.get(this.env.RUNNER.idFromName(p.container));
1216+ await stub.startJob({ repoId: this.repo.meta.repoId, jobId: p.jobId, container: p.container, token: p.token, apiUrl: this.env.PUBLIC_URL, egress: p.egress });
11641217 } catch (e) {
11651218 const message = (e as Error).message;
1166- if (RETRYABLE_START.test(message)) return this.placementFailed(jobId, message);
1167- await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: `Could not start a runner: ${message}` } });
1219+ if (RETRYABLE_START.test(message)) return this.placementFailed(p.jobId, message);
1220+ await this.completeJob(p.jobId, { conclusion: "failure", outputs: {}, failure: { message: `Could not start a runner: ${message}` } });
11681221 }
11691222 }
11701223
modifiedapps/diffinery/src/runner/containers.ts+19−7View file
@@ -37,6 +37,10 @@
3737 }
3838 }
3939
40+/** How long a job's container may take to come up, and how often it is checked meanwhile. */
41+const START_WAIT_MS = 180_000;
42+const START_POLL_MS = 500;
43+
4044 /**
4145 * One container per workflow job. The runner agent inside fetches its job spec
4246 * from the Diffinery API, runs the steps and reports back, then exits.
@@ -48,14 +52,22 @@
4852 async startJob(input: { repoId: string; jobId: string; container?: string; token: string; apiUrl: string; egress?: Egress }): Promise<void> {
4953 await this.ctx.storage.put("job", { repoId: input.repoId, jobId: input.jobId, container: input.container });
5054 const egressEnv = await this.useEgress(input.egress ?? "open");
51- await this.start({
52- envVars: {
53- DIFFINERY_API_URL: input.apiUrl,
54- DIFFINERY_JOB_ID: input.jobId,
55- DIFFINERY_JOB_TOKEN: input.token,
56- ...egressEnv,
55+ await this.start(
56+ {
57+ envVars: {
58+ DIFFINERY_API_URL: input.apiUrl,
59+ DIFFINERY_JOB_ID: input.jobId,
60+ DIFFINERY_JOB_TOKEN: input.token,
61+ ...egressEnv,
62+ },
5763 },
58- });
64+ // The library gives up after 8 s and reports "no container instance", but a container on a host
65+ // that hasn't pulled the runner image yet takes longer than that to come up: such jobs went back
66+ // in line for 30 to 90 s more (measured on 10-leg matrices). Wait for it instead.
67+ // The runner listens on no port: the library's own fallback port (33) answers "not listening" once it is up.
68+ // Each check can itself take up to 5 s, so the signal bounds the whole wait.
69+ { portToCheck: 33, waitInterval: START_POLL_MS, retries: Math.ceil(START_WAIT_MS / START_POLL_MS), signal: AbortSignal.timeout(START_WAIT_MS + 60_000) },
70+ );
5971 }
6072
6173 async stopJob(): Promise<void> {
modifiedapps/diffinery/src/web/code.tsx+24−0View file
@@ -119,6 +119,26 @@
119119 );
120120 };
121121
122+const NoDefaultBranch: FC<{ base: string; branch: string; names: { branches: string[]; tags: string[] } }> = ({ base, branch, names }) => (
123+ <div class="box" style="max-width:900px">
124+ <div class="head">
125+ <h2>No {branch} branch yet</h2>
126+ <span class="muted small">
127+ The default branch, <code>{branch}</code>, hasn't been pushed. Other branches have:
128+ </span>
129+ </div>
130+ <div class="body">
131+ <ul class="m0">
132+ {names.branches.slice(0, 100).map((b) => (
133+ <li>
134+ <a href={`${base}/tree/${encPath(b)}`}>{b}</a>
135+ </li>
136+ ))}
137+ </ul>
138+ </div>
139+ </div>
140+);
141+
122142 async function treePage(c: AppContext, l: Loaded, ref: string, path: string, root: boolean): Promise<Response> {
123143 const { repo, caller, stub } = l;
124144 const base = repoUrl(repo);
@@ -134,6 +154,10 @@
134154 if (!names.branches.length && !names.tags.length) {
135155 return render(c, { title: `${repo.owner_login}/${repo.name}`, repo: { repo, caller, tab: "code", counts } }, <QuickSetup c={c} l={l} />);
136156 }
157+ // Branches were pushed, but not the default one: say so and list them rather than answer 404.
158+ if (root && !path && ref === repo.default_branch) {
159+ return render(c, { title: `${repo.owner_login}/${repo.name}`, repo: { repo, caller, tab: "code", counts } }, <NoDefaultBranch base={base} branch={ref} names={names} />);
160+ }
137161 const blob = path ? await stub.blob(ref, path).catch(() => null) : null;
138162 if (blob) return c.redirect(`${base}/blob/${encPath(ref)}/${encPath(path)}`, 302);
139163 throw new HttpError(404, "path not found");
addedapps/diffinery/test/unit/launch.test.ts+158−0View file
@@ -1,0 +1,158 @@
1+import { describe, expect, it } from "vitest";
2+import { DatabaseSync } from "node:sqlite";
3+import { parseWorkflow } from "@diffinery/workflow-yaml";
4+import { Actions, MAX_HOSTED_JOBS } from "../../src/repo/actions.js";
5+import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
6+import { testD1 } from "./stubs/d1.js";
7+import type { RepoDO } from "../../src/repo/RepoDO.js";
8+import type { Env } from "../../src/env.js";
9+import type { Caller } from "../../src/repo/types.js";
10+
11+/** The Durable Object's SqlStorage over node:sqlite. */
12+function sqlStorage() {
13+ const db = new DatabaseSync(":memory:");
14+ return {
15+ exec(query: string, ...args: unknown[]) {
16+ const params = args.map((a) => (a === undefined ? null : a)) as Array<string | number | null>;
17+ let rows: unknown[] = [];
18+ if (!params.length && query.trim().replace(/;\s*$/, "").includes(";")) db.exec(query);
19+ else rows = db.prepare(query).all(...params);
20+ return {
21+ toArray: () => rows as never[],
22+ one: () => {
23+ if (rows.length !== 1) throw new Error(`expected one row, got ${rows.length}`);
24+ return rows[0] as never;
25+ },
26+ };
27+ },
28+ };
29+}
30+
31+const ADA: Caller = { id: "u1", login: "ada", kind: "user", role: "write" };
32+
33+const matrix = (legs: number) => `name: w
34+on: push
35+jobs:
36+ test:
37+ runs-on: ubuntu-latest
38+ strategy:
39+ matrix:
40+ n: [${Array.from({ length: legs }, (_, i) => i + 1).join(", ")}]
41+ steps: [{ run: echo test }]
42+`;
43+
44+type Row = { id: string; status: string; fleet_held: number; placement_tries: number; container: string | null };
45+
46+/**
47+ * Actions over an in-memory repository and D1, with a runner stand-in whose starts take `startMs`
48+ * and fail for the containers `refuse` names.
49+ */
50+function fixture(o: { startMs?: number; refuse?: (n: string) => string | null } = {}) {
51+ const sql = sqlStorage();
52+ sql.exec(REPO_SCHEMA);
53+ migrateRepoSchema(sql as unknown as SqlStorage);
54+ const DB = testD1();
55+ DB.sqlite.prepare("INSERT INTO accounts (id, login, kind, status, created_at) VALUES ('acme', 'acme', 'org', 'active', ?)").run(new Date().toISOString());
56+ const started: string[] = [];
57+ let active = 0;
58+ let most = 0;
59+ const events: Array<{ type: string; payload: unknown }> = [];
60+ let alarms = 0;
61+ const repo = {
62+ sql,
63+ meta: { repoId: "r", owner: "acme", ownerId: "acme", ownerKind: "org", name: "app", defaultBranch: "main" },
64+ headRef: "refs/heads/main",
65+ getRef: () => null,
66+ upsertCheck: () => {},
67+ broadcast: () => {},
68+ emit: () => {},
69+ armAlarm: () => void alarms++,
70+ deploys: { jobCompleted: () => {}, protection: () => null },
71+ runtime: { sweepJob: async () => {} },
72+ policy: { agentOrigin: () => false },
73+ } as unknown as RepoDO;
74+ const env = {
75+ DB,
76+ PUBLIC_URL: "https://d.test",
77+ BUCKET: { put: async () => {}, get: async () => null },
78+ RUNNER: {
79+ idFromName: (n: string) => n,
80+ get: (n: string) => ({
81+ startJob: async () => {
82+ active++;
83+ most = Math.max(most, active);
84+ await new Promise((r) => setTimeout(r, o.startMs ?? 5));
85+ active--;
86+ const refused = o.refuse?.(n);
87+ if (refused) throw new Error(refused);
88+ started.push(n);
89+ },
90+ stopJob: async () => {},
91+ }),
92+ },
93+ RUN_WORKFLOW: { create: async () => {}, get: async () => ({ sendEvent: async (e: { type: string; payload: unknown }) => void events.push(e) }) },
94+ } as unknown as Env;
95+ const actions = new Actions(repo, env);
96+ const jobs = () => sql.exec("SELECT * FROM jobs ORDER BY rowid").toArray() as Row[];
97+ async function start(legs: number) {
98+ const { workflow } = parseWorkflow(matrix(legs), ".github/workflows/w.yml");
99+ Object.defineProperty(actions, "loadWorkflows", { value: async () => [{ path: ".github/workflows/w.yml", workflow, errors: [] }] });
100+ const run = (await actions.createRun(workflow!, { event: "push", ref: "refs/heads/main", sha: "abc", payload: {}, caller: ADA, title: "t" }))!;
101+ expect((await actions.plan(run.id)).error).toBeUndefined();
102+ return actions.startJobKey(run.id, "test");
103+ }
104+ return { DB, sql, actions, started, jobs, start, events, most: () => most, alarms: () => alarms };
105+}
106+
107+describe("starting a matrix's hosted jobs", () => {
108+ it("starts several containers at once, up to the repository's share, and holds the rest", async () => {
109+ const f = fixture({ startMs: 20 });
110+ f.DB.sqlite.prepare("INSERT INTO account_quotas (account_id, hosted_jobs, updated_at, updated_by) VALUES ('acme', -1, ?, 'test')").run(new Date().toISOString());
111+ const legs = MAX_HOSTED_JOBS + 6;
112+ const t = Date.now();
113+ const { jobIds } = await f.start(legs);
114+ expect(jobIds).toHaveLength(legs);
115+ expect(f.started).toHaveLength(MAX_HOSTED_JOBS);
116+ // Not one after another: 20 starts of 20 ms each took far less than 400 ms.
117+ expect(f.most()).toBeGreaterThan(1);
118+ expect(f.most()).toBeLessThanOrEqual(8);
119+ expect(Date.now() - t).toBeLessThan(MAX_HOSTED_JOBS * 20);
120+ const rows = f.jobs();
121+ expect(rows.filter((j) => j.status === "waiting")).toHaveLength(MAX_HOSTED_JOBS);
122+ expect(rows.filter((j) => j.status === "queued" && j.fleet_held === 1)).toHaveLength(6);
123+ });
124+
125+ it("holds the legs past the account's hosted jobs for its alarm, asking the quota once for them", async () => {
126+ const f = fixture();
127+ f.DB.sqlite.prepare("INSERT INTO account_quotas (account_id, hosted_jobs, updated_at, updated_by) VALUES ('acme', 3, ?, 'test')").run(new Date().toISOString());
128+ let quotaReads = 0;
129+ const prepare = f.DB.prepare.bind(f.DB);
130+ (f.DB as unknown as { prepare: typeof prepare }).prepare = (q: string) => {
131+ if (q.includes("FROM account_quotas")) quotaReads++;
132+ return prepare(q);
133+ };
134+ await f.start(10);
135+ expect(f.started).toHaveLength(3);
136+ const held = f.jobs().filter((j) => j.status === "queued" && j.fleet_held === 2);
137+ expect(held).toHaveLength(7);
138+ // Three admitted, the fourth told to wait; the other six joined it without asking again.
139+ expect(quotaReads).toBe(4);
140+ expect(f.alarms()).toBeGreaterThan(0);
141+ });
142+
143+ it("puts a leg that got no machine back in line without holding up the others", async () => {
144+ let refused = false;
145+ const f = fixture({
146+ refuse: (n) => {
147+ if (refused || n !== `job-${f.jobs()[1]!.id}`) return null;
148+ refused = true;
149+ return "there is no container instance that can be provided to this durable object";
150+ },
151+ });
152+ await f.start(4);
153+ const rows = f.jobs();
154+ expect(f.started).toHaveLength(3);
155+ expect(rows[1]).toMatchObject({ status: "queued", placement_tries: 1, container: null });
156+ expect(f.events).toEqual([{ type: `job-${rows[1]!.id}`, payload: { retryInSeconds: 30 } }]);
157+ });
158+});
modifiedapps/diffinery/wrangler.jsonc+3−1View file
@@ -89,7 +89,9 @@
8989 // job's container seconds after it started. The rollout's SIGTERM reaches every process in the
9090 // container, so the job's steps died with it.
9191 "containers": [
92- { "class_name": "RunnerContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-2", "max_instances": 100, "rollout_active_grace_period": 23400, "rollout_step_percentage": 100 },
92+ // standard-4 (4 vCPU, 12 GiB, 20 GB): standard-2 has 1 vCPU, on which typecheck and test steps took
93+ // 2.3 to 2.6 times as long. Active CPU is what's billed, so a faster job costs about the same.
94+ { "class_name": "RunnerContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-4", "max_instances": 100, "rollout_active_grace_period": 23400, "rollout_step_percentage": 100 },
9395 { "class_name": "SandboxContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-1", "max_instances": 50, "rollout_active_grace_period": 3600, "rollout_step_percentage": 100 },
9496 // GitHub imports: a mirror clone of a large repository needs disk and memory.
9597 { "class_name": "ImportContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-4", "max_instances": 20, "rollout_active_grace_period": 25200, "rollout_step_percentage": 100 }