Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Hosted runners in four sizes, each job sized by what its own runs used (#198)

* Hosted runners in four sizes, each job sized by what its earlier runs used (WIP)
* Runner sizes: a job that waited for a machine starts large; tests
* Runner sizes: at least three measured runs before a job gets a smaller runner
* Runner sizes: a smaller runner must cover a job's busiest moment, not its average
* Runner sizes: CPU headroom, and back to large when a smaller runner was slower
test-bot authored 1 parent 560b60ecommit 8994af4b8478912d2d3a5a9b16a2279fa6e42df4Browse files
20 files changed+896−19
Files (20)
modifiedapps/diffinery/docs/actions/runners.md+23−1View file
@@ -62,7 +62,7 @@
6262
6363 | Property | Value |
6464 |---|---|
65-| Platform | Cloudflare Containers, instance type `standard-4`: 4 vCPU, 12 GiB of memory, 20 GB of disk |
65+| Platform | Cloudflare Containers. A large runner (instance type `standard-4`) has 4 vCPU, 12 GiB of memory and 20 GB of disk; see [Runner sizes](#runner-sizes) for the smaller ones |
6666 | Architecture | `linux/amd64` (`runner.arch` is `X64`) |
6767 | Operating system | Ubuntu 24.04 (`runner.os` is `Linux`) |
6868 | User | `root`. `sudo` is installed and needs no password, so `sudo apt-get install ...` works as on GitHub |
@@ -73,6 +73,28 @@
7373 | Locale | `LANG=C.UTF-8`, `LC_ALL=C.UTF-8` |
7474 | Concurrency | Up to 100 hosted job containers run at once across diffinery.com, and at most 20 from one repository; its further jobs stay queued until one of its own finishes. See [Limits](/docs/reference/limits) |
7575
76+### Runner sizes
77+
78+Hosted runners come in four sizes. Every size runs the same image, so a job behaves the same on each; only the machine differs.
79+
80+| Size | `runs-on` label | vCPU | Memory | Disk | Cloudflare instance type |
81+|---|---|---|---|---|---|
82+| large | `diffinery-large` | 4 | 12 GiB | 20 GB | `standard-4` |
83+| medium | `diffinery-medium` | 2 | 8 GiB | 16 GB | `standard-3` |
84+| small | `diffinery-small` | 1/2 | 4 GiB | 8 GB | `standard-1` |
85+| tiny | `diffinery-tiny` | 1/4 | 1 GiB | 4 GB | `basic` |
86+
87+A job with one of these labels in `runs-on` gets that size (with several, the largest): `runs-on: [ubuntu-latest, diffinery-small]`, or the label alone. A job without one gets **large**, unless its own earlier runs show a smaller size serves it just as well. That's how Diffinery keeps hosted minutes cheap: Cloudflare bills a container's memory and disk for what its size provides, used or not, so a deploy job that waits on another service for a minute costs a large runner's memory for that minute.
88+
89+How a size is picked for a job without a label:
90+
91+- The runner measures every hosted job: busy cores every 2 seconds, memory and disk in use every half second, and processes the kernel killed for want of memory. The last lines of the job's log say what it used, for example `Resources: 0.1 of 4 vCPUs busy on average (0.3 at the 95th percentile), peak memory 0.3 GiB of 12.0 GiB, peak disk 1.6 GB of 19.3 GB`. The job API (`GET /repos/{owner}/{repo}/actions/jobs/{job_id}`) returns it as `usage`, and the size as `runner_size`.
92+- The next run of the same job (the same workflow file and job name, so each matrix leg on its own) looks at up to its last 5 runs, and needs at least 3. It gets a smaller size only if every one of them would have fit it with room to spare: peak memory times 1.5, plus 0.5 GiB, and peak disk times 1.25, plus 1 GB, within the size; and at least half as many vCPUs again as the job kept busy in its busiest 2-second window, so a job that bursts onto several cores even briefly, such as a test suite, keeps a runner with room for that. A job that kept its runner's CPU busy to the limit never gets a smaller size than that one.
93+- A job stays on large when it has no measured runs yet, when one of those runs ran out of memory, when one failed or timed out on a smaller size, or when one took longer on a smaller size than the job typically takes on large (its median, plus a fifth and 10 seconds). A re-run always runs on large, and so does a job that found no smaller runner free when it first started. So a smaller size is never the reason a job fails or slows down twice.
94+- The job's log names the size it got and why, near the top: `Runner size: tiny (0.25 of a vCPU, 1 GiB of memory, 4 GB of disk), sized by this job's last 5 runs (peak memory 0.3 GiB)`.
95+
96+Set `runs-on: diffinery-large` to keep a job on large whatever it uses.
97+
7698 ### The Ubuntu 24.04 image
7799
78100 The hosted image is built from `containers/runner/Dockerfile` in the Diffinery source. It is deliberately small. This is the complete list of what is preinstalled:
modifiedapps/diffinery/docs/reference/limits.md+2−2View file
@@ -211,7 +211,7 @@
211211 | Schedule granularity | 5 minutes | Scheduled workflows are checked every 5 minutes. A cron that fires more often still runs at most once per 5-minute window. Only workflows on the default branch of repositories that aren't archived are scheduled. |
212212 | Schedule catch-up | 60 minutes | A check that comes late, or fails, is caught up by the next one, up to an hour back. Each schedule starts once per check, however many of its minutes the check covers. |
213213 | Starting a job | 3 retries | Starting a job's container is retried 3 times with exponential backoff from 10 seconds before the job fails with "Could not start a runner". |
214-| Hosted runner size | Cloudflare Containers `standard-4`: 4 vCPU, 12 GiB of memory, 20 GB of disk | Each `runs-on: ubuntu-latest` job runs in its own container of this instance type. |
214+| Hosted runner sizes | large: `standard-4` (4 vCPU, 12 GiB of memory, 20 GB of disk); medium: `standard-3` (2 vCPU, 8 GiB, 16 GB); small: `standard-1` (1/2 vCPU, 4 GiB, 8 GB); tiny: `basic` (1/4 vCPU, 1 GiB, 4 GB) | Each hosted job runs in its own container. A job runs on large unless `runs-on` names a size or its earlier runs show a smaller one fits with headroom ([Runner sizes](/docs/actions/runners#runner-sizes)). |
215215 | Hosted job containers at once | 100 | `max_instances` of the runner container class. |
216216 | Hosted jobs at once per repository | 20 | Further jobs stay queued and start, oldest first, as the repository's own jobs finish, so one repository can't take every container. Self-hosted jobs don't count. |
217217 | No free container | Wait, then retry, 8 times | When the platform has no container free for a job (many jobs starting at once, or just after a deploy), the start fails on the platform's side in a way that passes ("Container sidecar is shutting down", a dropped connection), or the job's container stops before its runner fetched the job, the job goes back to queued and starts again after 30 seconds, then 1, 2, 4 and 5 minutes. After 8 tries, about half an hour, it fails with `No hosted runner could be started`. Only jobs whose runner never started are retried, so no step runs twice. |
@@ -378,7 +378,7 @@
378378 | --- | --- | --- |
379379 | Worker CPU time | 300,000 ms per request | `limits.cpu_ms`. |
380380 | Live update connection | 10 minutes, up to 10 topics | A page's WebSocket is closed with code `4001` after 10 minutes and reconnects at once, so access is checked again. |
381-| Runner containers | `standard-4`, up to 100 instances | `containers[RunnerContainer]`. |
381+| Runner containers | Up to 100 instances of each size: `standard-4`, `standard-3`, `standard-1` and `basic` | `containers[RunnerContainer]`, `[RunnerContainerMedium]`, `[RunnerContainerSmall]`, `[RunnerContainerTiny]`. |
382382 | Sandbox containers | `standard-1`, up to 50 instances | `containers[SandboxContainer]`. |
383383 | Import containers | `standard-4`, up to 20 instances | `containers[ImportContainer]`: one container per Git transfer of an import. |
384384 | Container rollout grace period | 23,400 seconds for runners, 25,200 for imports, 3,600 for sandboxes | `rollout_active_grace_period`. A deploy doesn't replace a container younger than this, so running jobs (6 hours at most) and import transfers finish first. |
modifiedapps/diffinery/docs/self-hosting/architecture.md+1−1View file
@@ -170,7 +170,7 @@
170170
171171 All three container classes use the same image, built from `containers/runner/Dockerfile` (Ubuntu 24.04 with Node.js 22, git, git-lfs, Python and build tools, plus the runner agent).
172172
173-- **`RunnerContainer`** runs one hosted job (`runs-on: ubuntu-latest` and other `ubuntu-*` or `linux` labels). The Durable Object starts a container named after the job with three environment variables: `DIFFINERY_API_URL`, `DIFFINERY_JOB_ID` and `DIFFINERY_JOB_TOKEN`. The agent fetches its job spec from `/runner/jobs/<id>`, runs the steps, streams logs, uses the cache and artifact services and the OIDC endpoint, reports completion and exits. If the container stops before the job reports completion, the job fails with "The runner stopped unexpectedly". Instance type `standard-4` (4 vCPU, 12 GiB), at most 100 at once, internet access on (only HTTP and HTTPS, through the Worker, under [`EGRESS=web`](/docs/self-hosting/configuration#egress)). Docker is started inside the container the first time a job uses it.
173+- **`RunnerContainer`** runs one hosted job (`runs-on: ubuntu-latest` and other `ubuntu-*` or `linux` labels). The Durable Object starts a container named after the job with three environment variables: `DIFFINERY_API_URL`, `DIFFINERY_JOB_ID` and `DIFFINERY_JOB_TOKEN`. The agent fetches its job spec from `/runner/jobs/<id>`, runs the steps, streams logs, uses the cache and artifact services and the OIDC endpoint, reports completion and exits. If the container stops before the job reports completion, the job fails with "The runner stopped unexpectedly". Instance type `standard-4` (4 vCPU, 12 GiB), at most 100 at once; `RunnerContainerMedium` (`standard-3`), `RunnerContainerSmall` (`standard-1`) and `RunnerContainerTiny` (`basic`) are the same runner on smaller instance types, at most 100 each, for jobs that ask for a size or whose earlier runs fit one ([Runner sizes](/docs/actions/runners#runner-sizes)); a job's container name ends in `.medium`, `.small` or `.tiny` on those. A deployment without their bindings (`RUNNER_MEDIUM`, `RUNNER_SMALL`, `RUNNER_TINY`) runs every job on `RunnerContainer`. Each runs with internet access on (only HTTP and HTTPS, through the Worker, under [`EGRESS=web`](/docs/self-hosting/configuration#egress)). Docker is started inside the container the first time a job uses it.
174174 - **`SandboxContainer`** runs one agent sandbox: the repository checked out at a ref, with an HTTP API on port 8080 for `exec`, file reads and file writes. The Worker forwards REST and MCP sandbox calls to it with a per-sandbox secret. Instance type `standard-1`, at most 50 at once, stopped after 30 minutes without requests. Internet access is on, limited to HTTP and HTTPS under `EGRESS=web` like `RunnerContainer`, and a limited sandbox stays limited when it starts again after its idle stop.
175175 - **`ImportContainer`** runs the Git transfer of one GitHub import: the runner agent in import mode, started with `DIFFINERY_API_URL`, `DIFFINERY_IMPORT_ID` and `DIFFINERY_IMPORT_TOKEN`. One container per try, named `<import id>-<attempt>-<try>`. If it stops before the agent reports a result (a crash, running out of memory, a platform restart), the transfer counts as failed and the import's Workflow can try it again. Instance type `standard-4` (a mirror clone of a large repository needs disk and memory), at most 20 at once, internet access on, stopped after 7 hours without activity.
176176
modifiedapps/diffinery/docs/self-hosting/configuration.md+1−1View file
@@ -210,7 +210,7 @@
210210 | `observability` | Enabled, `head_sampling_rate: 1` | Every request is logged to Workers Logs. |
211211 | `upload_source_maps` | `true` | Readable stack traces in the logs. |
212212 | `limits.cpu_ms` | `300000` | Up to 5 minutes of CPU time per request (large pushes, diffs and merges). |
213-| `containers` | `RunnerContainer`: `standard-4`, `max_instances: 100`; `SandboxContainer`: `standard-1`, `max_instances: 50`; `ImportContainer`: `standard-4`, `max_instances: 20`; `rollout_active_grace_period`: 23400 for runners and 25200 for imports (longer than a job or transfer can run, so a deploy never replaces a running one), 3600 for sandboxes; `rollout_step_percentage: 100`, so containers started after a deploy get the new image | All three build `containers/runner/Dockerfile` with `containers/runner` as context. Raise `max_instances` for more concurrent jobs, sandboxes or import transfers. |
213+| `containers` | `RunnerContainer`: `standard-4`, `max_instances: 100`; `RunnerContainerMedium`, `RunnerContainerSmall`, `RunnerContainerTiny`: `standard-3`, `standard-1`, `basic`, `max_instances: 100` each (smaller hosted runners; remove them and their `RUNNER_*` bindings to run every job on `standard-4`); `SandboxContainer`: `standard-1`, `max_instances: 50`; `ImportContainer`: `standard-4`, `max_instances: 20`; `rollout_active_grace_period`: 23400 for runners and 25200 for imports (longer than a job or transfer can run, so a deploy never replaces a running one), 3600 for sandboxes; `rollout_step_percentage: 100`, so containers started after a deploy get the new image | All three build `containers/runner/Dockerfile` with `containers/runner` as context. Raise `max_instances` for more concurrent jobs, sandboxes or import transfers. |
214214 | `triggers.crons` | `["*/5 * * * *"]` | Token expiry reminders, email log pruning, registry cleanup, cancelling draft imports older than 7 days, and setting the workflow scheduler's alarm if it isn't set. Scheduled workflows run from that alarm, not from the cron trigger. |
215215
216216 ## The dev environment
modifiedapps/diffinery/src/env.ts+4−0View file
@@ -24,6 +24,10 @@
2424 /** End-to-end tests only: how often open web terminals are checked, in ms (default 60000, at least 2000). */
2525 TERMINAL_POLL_MS?: string;
2626 RUNNER: DurableObjectNamespace<RunnerContainer>;
27+ /** Smaller hosted runners (runner/sizes.ts); a deployment without one runs those jobs on RUNNER. */
28+ RUNNER_MEDIUM?: DurableObjectNamespace<RunnerContainer>;
29+ RUNNER_SMALL?: DurableObjectNamespace<RunnerContainer>;
30+ RUNNER_TINY?: DurableObjectNamespace<RunnerContainer>;
2731 SANDBOX: DurableObjectNamespace<SandboxContainer>;
2832 /** One per repository: its agent tasks (agentic/ProjectDO.ts). */
2933 PROJECT: DurableObjectNamespace<ProjectDO>;
modifiedapps/diffinery/src/http/gh.ts+3−0View file
@@ -652,6 +652,7 @@
652652 startedAt: string | null;
653653 completedAt: string | null;
654654 runnerName?: string | null;
655+ runnerSize?: string | null;
655656 steps: Array<{ index: number; name: string; status: string; conclusion?: string | null; startedAt?: string; completedAt?: string }>;
656657 };
657658
@@ -676,6 +677,8 @@
676677 check_run_url: `${api}/check-runs/${j.id}`,
677678 labels: [] as string[],
678679 runner_name: j.runnerName ?? null,
680+ // Hosted runner size (large, medium, small, tiny); `usage` (from ...j) is what the job used of it.
681+ runner_size: j.runnerSize ?? null,
679682 steps: j.steps.map((s) => ({ ...s, number: s.index + 1, started_at: isoSeconds(s.startedAt), completed_at: isoSeconds(s.completedAt), conclusion: s.conclusion ?? null })),
680683 };
681684 }
modifiedapps/diffinery/src/index.ts+1−1View file
@@ -7,7 +7,7 @@
77
88 export { RepoDO } from "./repo/RepoDO.js";
99 export { SchedulerDO } from "./scheduler.js";
10-export { RunnerContainer, SandboxContainer, SandboxPoolDO } from "./runner/containers.js";
10+export { RunnerContainer, RunnerContainerMedium, RunnerContainerSmall, RunnerContainerTiny, SandboxContainer, SandboxPoolDO } from "./runner/containers.js";
1111 export { RunWorkflow } from "./runner/workflow.js";
1212 export { ImportWorkflow } from "./imports/workflow.js";
1313 export { AccountWorkflow } from "./accounts/workflow.js";
modifiedapps/diffinery/src/repo/actions.ts+80−10View file
@@ -1,5 +1,5 @@
11 import { ZERO_OID, diffTrees, fromUtf8, isTreeMode, readCommit, readTree, resolvePath } from "@diffinery/git-core";
2-import type { Actor, Conclusion, Json, JobCompletion, JobSpec, LogLine, NeedsResult, StepReport } from "@diffinery/shared";
2+import type { Actor, Conclusion, Json, JobCompletion, JobSpec, JobUsage, LogLine, NeedsResult, StepReport } from "@diffinery/shared";
33 import {
44 evaluateIf,
55 evaluateValue,
@@ -37,6 +37,7 @@
3737 import { schedulerStub } from "../lib/schedules.js";
3838 import { APPROVAL_DAYS, type Deployment, type DeploymentReview, type Reviewer } from "./deployments.js";
3939 import { referencedSecrets, RUNNER_SECRETS, secretsFor } from "./job-secrets.js";
40+import { chooseSize, cleanUsage, containerName, describeSize, RUNNER_SIZES, sizeLabel, sizeOfContainer, type RunnerSize, type SizedRun } from "../runner/sizes.js";
4041
4142 /** How long a sandbox token works after the sandbox starts. */
4243 export const SANDBOX_TOKEN_TTL_MS = 24 * 3600_000;
@@ -93,7 +94,7 @@
9394 const RUNNER_ONLINE_MS = 90_000;
9495
9596 /** A hosted job admitted to start (Actions.admit), and what its container starts with. */
96-type Placement = { jobId: string; container: string; token: string; egress: Egress };
97+type Placement = { jobId: string; container: string; token: string; egress: Egress; vcpu: number };
9798
9899 type RunnerRow = { id: number; name: string; labels: string; token_hash: string | null; ephemeral: number; created_by: string; created_at: string; last_seen_at: string | null };
99100
@@ -204,6 +205,12 @@
204205 /** Set on a job a partial re-run copied from an earlier attempt: the job that actually ran. */
205206 carried_from?: string | null;
206207 restarts: number;
208+ /** Hosted runner size runs-on asked for (runner/sizes.ts); null when it is chosen from the job's earlier runs. */
209+ runner_size?: string | null;
210+ /** The log line naming the size the job got and why. */
211+ size_note?: string | null;
212+ /** What the job used of its hosted runner (JSON JobUsage), as the runner measured it. */
213+ usage?: string | null;
207214 };
208215
209216 /** Where a job ran, as its runner reported it: the runner image, the container's location and instance. */
@@ -281,6 +288,10 @@
281288 placementTries: number;
282289 /** Times the platform replaced the job's runner before its steps ran, and the job started again. */
283290 restarts: number;
291+ /** A hosted job's runner size (runner/sizes.ts); null for self-hosted jobs and jobs not placed yet. */
292+ runnerSize: RunnerSize | null;
293+ /** What the job used of its hosted runner, as the runner measured it. */
294+ usage: JobUsage | null;
284295 };
285296
286297 const toRun = (r: RunRow): RunInfo => ({
@@ -322,6 +333,8 @@
322333 carriedFrom: j.carried_from ?? null,
323334 placementTries: j.placement_tries ?? 0,
324335 restarts: j.restarts ?? 0,
336+ runnerSize: j.container && !j.container.startsWith("self-hosted:") ? sizeOfContainer(j.container) : null,
337+ usage: j.usage ? (JSON.parse(j.usage) as JobUsage) : null,
325338 });
326339
327340 /**
@@ -1080,7 +1093,7 @@
10801093 }
10811094 const id = newId();
10821095 this.sql.exec(
1083- "INSERT INTO jobs (id, run_id, job_key, name, status, matrix, strategy, timeout_minutes, labels, continue_on_error, concurrency_group, concurrency_cancel, environment) VALUES (?, ?, ?, ?, 'queued', ?, ?, ?, ?, ?, ?, ?, ?)",
1096+ "INSERT INTO jobs (id, run_id, job_key, name, status, matrix, strategy, timeout_minutes, labels, continue_on_error, concurrency_group, concurrency_cancel, environment, runner_size) VALUES (?, ?, ?, ?, 'queued', ?, ?, ?, ?, ?, ?, ?, ?, ?)",
10841097 id,
10851098 runId,
10861099 jobKey,
@@ -1093,6 +1106,7 @@
10931106 leg.group,
10941107 leg.cancel ? 1 : 0,
10951108 leg.environment,
1109+ selfHosted ? null : sizeLabel(leg.labels),
10961110 );
10971111 ids.push(id);
10981112 });
@@ -1218,19 +1232,72 @@
12181232 }
12191233 this.sql.exec("UPDATE jobs SET fleet_held = 0 WHERE id = ?", jobId);
12201234 const token = `djt_${m.repoId}_${jobId}_${randomToken(24)}`;
1235+ const { size, note } = this.runnerSize(job, run);
12211236 // A restarted job gets a new container: the replaced one may still be shutting down.
1222- const container = job.restarts ? `job-${jobId}-r${job.restarts}` : `job-${jobId}`;
1223- this.sql.exec("UPDATE jobs SET status = 'waiting', token_hash = ?, container = ? WHERE id = ?", await sha256(token), container, jobId);
1237+ const container = containerName(jobId, job.restarts, size);
1238+ this.sql.exec("UPDATE jobs SET status = 'waiting', token_hash = ?, container = ?, size_note = ? WHERE id = ?", await sha256(token), container, note, jobId);
12241239 await startCompute(this.env, { id: jobId, accountId: m.ownerId, repoId: m.repoId, kind: "job" }).catch((e: unknown) => console.error("compute usage failed", e));
12251240 this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
1226- return { jobId, container, token, egress: await egressFor(this.env, m.ownerId) };
1241+ return { jobId, container, token, egress: await egressFor(this.env, m.ownerId), vcpu: RUNNER_SIZES[size].vcpu };
12271242 }
12281243
1244+ /**
1245+ * The hosted runner size for a job (runner/sizes.ts): what its runs-on asks for, or else what its own
1246+ * earlier runs (same workflow file and job name, on hosted runners) show it needs. A size this
1247+ * deployment has no container class for runs as large.
1248+ */
1249+ private runnerSize(job: JobRow, run: RunRow): { size: RunnerSize; note: string } {
1250+ const pinned = (job.runner_size as RunnerSize | null | undefined) ?? null;
1251+ let size: RunnerSize;
1252+ let why: string;
1253+ if (pinned) {
1254+ size = pinned;
1255+ why = "asked for by runs-on";
1256+ } else {
1257+ const rows = this.sql
1258+ .exec<{ conclusion: string | null; container: string; usage: string | null }>(
1259+ `SELECT j.conclusion, j.container, j.usage FROM jobs j JOIN runs r ON r.id = j.run_id
1260+ WHERE r.workflow_path = ? AND j.name = ? AND j.id != ? AND j.status = 'completed' AND j.labels IS NULL AND j.container IS NOT NULL AND j.carried_from IS NULL
1261+ ORDER BY j.completed_at DESC LIMIT 20`,
1262+ run.workflow_path,
1263+ job.name,
1264+ job.id,
1265+ )
1266+ .toArray();
1267+ const history: SizedRun[] = rows.map((r) => {
1268+ let usage: SizedRun["usage"] = null;
1269+ try {
1270+ usage = r.usage ? JSON.parse(r.usage) : null;
1271+ } catch {
1272+ /* unreadable: counts as not measured */
1273+ }
1274+ return { conclusion: r.conclusion, size: sizeOfContainer(r.container), usage };
1275+ });
1276+ ({ size, why } = chooseSize(history, { rerun: run.attempt > 1 }));
1277+ // A job that already waited for a machine doesn't wait again for a smaller one.
1278+ if (size !== "large" && job.placement_tries > 0) {
1279+ size = "large";
1280+ why = "no smaller runner was free when it first started";
1281+ }
1282+ }
1283+ if (size !== "large" && !this.env[RUNNER_SIZES[size].binding]) {
1284+ size = "large";
1285+ why = "this Diffinery has no smaller runners";
1286+ }
1287+ return { size, note: describeSize(size, why, !!pinned && size === pinned) };
1288+ }
1289+
1290+ /** The container class of a hosted job's container (its name says which size it is). */
1291+ private runnerNamespace(container: string) {
1292+ return this.env[RUNNER_SIZES[sizeOfContainer(container)].binding] ?? this.env.RUNNER;
1293+ }
1294+
12291295 /** Starts an admitted job's container. One that got no machine goes back in line (placementFailed). */
12301296 private async place(p: Placement): Promise<void> {
12311297 try {
1232- const stub = this.env.RUNNER.get(this.env.RUNNER.idFromName(p.container));
1233- await stub.startJob({ repoId: this.repo.meta.repoId, jobId: p.jobId, container: p.container, token: p.token, apiUrl: this.env.PUBLIC_URL, egress: p.egress });
1298+ const ns = this.runnerNamespace(p.container);
1299+ const stub = ns.get(ns.idFromName(p.container));
1300+ await stub.startJob({ repoId: this.repo.meta.repoId, jobId: p.jobId, container: p.container, token: p.token, apiUrl: this.env.PUBLIC_URL, egress: p.egress, vcpu: p.vcpu });
12341301 } catch (e) {
12351302 const message = (e as Error).message;
12361303 if (RETRYABLE_START.test(message)) return this.placementFailed(p.jobId, message);
@@ -1628,7 +1695,8 @@
16281695 private async stopContainer(j: JobRow): Promise<void> {
16291696 if (!j.container || j.container.startsWith("self-hosted:")) return;
16301697 try {
1631- await this.env.RUNNER.get(this.env.RUNNER.idFromName(j.container)).stopJob();
1698+ const ns = this.runnerNamespace(j.container);
1699+ await ns.get(ns.idFromName(j.container)).stopJob();
16321700 } catch (e) {
16331701 console.warn("stop container failed", e);
16341702 }
@@ -1978,6 +2046,7 @@
19782046 this.repo.runtime.grant(jobId, permissions);
19792047 const idToken = permissions["id-token"] === "write";
19802048 return {
2049+ ...(job.size_note && !job.labels ? { runnerSize: job.size_note } : {}),
19812050 runId: run.id,
19822051 jobId,
19832052 jobKey: short,
@@ -2121,13 +2190,14 @@
21212190 const t = now();
21222191 const conclusion = override ?? completion.conclusion;
21232192 this.sql.exec(
2124- "UPDATE jobs SET status = 'completed', conclusion = ?, completed_at = ?, outputs = ?, failure = ?, summary = ?, token_hash = NULL, started_at = COALESCE(started_at, ?) WHERE id = ?",
2193+ "UPDATE jobs SET status = 'completed', conclusion = ?, completed_at = ?, outputs = ?, failure = ?, summary = ?, token_hash = NULL, started_at = COALESCE(started_at, ?), usage = COALESCE(?, usage) WHERE id = ?",
21252194 conclusion,
21262195 t,
21272196 JSON.stringify(completion.outputs ?? {}),
21282197 completion.failure ? JSON.stringify(completion.failure) : null,
21292198 completion.summary ?? null,
21302199 t,
2200+ job.labels ? null : cleanUsage(completion.usage),
21312201 jobId,
21322202 );
21332203 this.repo.deploys.jobCompleted(jobId, conclusion);
modifiedapps/diffinery/src/repo/schema.ts+5−0View file
@@ -201,6 +201,11 @@
201201 ["jobs", "runner_info", "TEXT"],
202202 // Times the platform replaced a hosted job's runner before its steps ran and the job started again (repo/actions.ts MAX_RUNNER_RESTARTS).
203203 ["jobs", "restarts", "INTEGER NOT NULL DEFAULT 0"],
204+ // Hosted runner size (runner/sizes.ts): the one runs-on asks for (null: chosen from the job's earlier runs),
205+ // the log line saying which size the job got and why, and what the job used of its runner (JSON JobUsage).
206+ ["jobs", "runner_size", "TEXT"],
207+ ["jobs", "size_note", "TEXT"],
208+ ["jobs", "usage", "TEXT"],
204209 // The GitHub import that wrote this row (null for everything else); a retry may only overwrite its own rows.
205210 ["issues", "imported", "TEXT"],
206211 ["pulls", "imported", "TEXT"],
modifiedapps/diffinery/src/runner/containers.ts+11−1View file
@@ -49,7 +49,7 @@
4949 // Jobs make no inbound requests, so activity-based sleep must not stop them.
5050 override sleepAfter = "6h";
5151
52- async startJob(input: { repoId: string; jobId: string; container?: string; token: string; apiUrl: string; egress?: Egress }): Promise<void> {
52+ async startJob(input: { repoId: string; jobId: string; container?: string; token: string; apiUrl: string; egress?: Egress; vcpu?: number }): Promise<void> {
5353 await this.ctx.storage.put("job", { repoId: input.repoId, jobId: input.jobId, container: input.container });
5454 const egressEnv = await this.useEgress(input.egress ?? "open");
5555 await this.start(
@@ -58,6 +58,8 @@
5858 DIFFINERY_API_URL: input.apiUrl,
5959 DIFFINERY_JOB_ID: input.jobId,
6060 DIFFINERY_JOB_TOKEN: input.token,
61+ // The runner's share of CPU, which the agent measures the job's use against (runner/sizes.ts).
62+ ...(input.vcpu ? { DIFFINERY_RUNNER_VCPU: String(input.vcpu) } : {}),
6163 ...egressEnv,
6264 },
6365 },
@@ -90,6 +92,14 @@
9092 }
9193
9294 /**
95+ * The smaller hosted runner sizes (runner/sizes.ts): the same runner on smaller Cloudflare instance
96+ * types, each its own container class because a container class has one instance type.
97+ */
98+export class RunnerContainerMedium extends RunnerContainer {}
99+export class RunnerContainerSmall extends RunnerContainer {}
100+export class RunnerContainerTiny extends RunnerContainer {}
101+
102+/**
93103 * Long-lived task sandbox for AI agents: the repo checked out with toolchains,
94104 * and an HTTP API (exec, read/write files) on port 8080.
95105 */
addedapps/diffinery/src/runner/sizes.ts+166−0View file
@@ -1,0 +1,166 @@
1+import { BUSY_BUCKETS, type JobUsage } from "@diffinery/shared";
2+
3+/**
4+ * Hosted runner sizes. Each is its own container class (wrangler.jsonc `containers`) on a Cloudflare
5+ * Containers instance type. Cloudflare bills a running container's memory and disk for what the
6+ * instance type provisions, used or not, and its CPU for what it actually uses; so a job that waits
7+ * on the network, or uses one core of four, costs most of a large runner's price in memory alone.
8+ *
9+ * A job runs on `large` unless it asks for a size (`runs-on: diffinery-medium`, ...) or its own
10+ * earlier runs show that a smaller one would serve it as well (chooseSize). Smaller is never meant to
11+ * be slower or less reliable: anything that could make it so keeps the job on `large`.
12+ */
13+export type RunnerSize = "large" | "medium" | "small" | "tiny";
14+export type RunnerBinding = "RUNNER" | "RUNNER_MEDIUM" | "RUNNER_SMALL" | "RUNNER_TINY";
15+
16+export const RUNNER_SIZES: Record<RunnerSize, { binding: RunnerBinding; instanceType: string; vcpu: number; memoryGiB: number; diskGB: number }> = {
17+ large: { binding: "RUNNER", instanceType: "standard-4", vcpu: 4, memoryGiB: 12, diskGB: 20 },
18+ medium: { binding: "RUNNER_MEDIUM", instanceType: "standard-3", vcpu: 2, memoryGiB: 8, diskGB: 16 },
19+ small: { binding: "RUNNER_SMALL", instanceType: "standard-1", vcpu: 0.5, memoryGiB: 4, diskGB: 8 },
20+ tiny: { binding: "RUNNER_TINY", instanceType: "basic", vcpu: 0.25, memoryGiB: 1, diskGB: 4 },
21+};
22+/** Smallest first: the order chooseSize tries them in. */
23+const ASCENDING: RunnerSize[] = ["tiny", "small", "medium", "large"];
24+
25+/** The size a `runs-on` label asks for (`diffinery-small`, ...), the largest if several; null for none. */
26+export function sizeLabel(labels: readonly string[]): RunnerSize | null {
27+ let out: RunnerSize | null = null;
28+ for (const l of labels) {
29+ const m = /^diffinery-(large|medium|small|tiny)$/i.exec(l.trim());
30+ if (m && (!out || ASCENDING.indexOf(m[1]!.toLowerCase() as RunnerSize) > ASCENDING.indexOf(out))) out = m[1]!.toLowerCase() as RunnerSize;
31+ }
32+ return out;
33+}
34+
35+/**
36+ * The container name of a hosted job: `job-<id>` (and `-r<n>` once restarted) on a large runner,
37+ * with `.<size>` after it on the others. The name says which container class runs it.
38+ */
39+export function containerName(jobId: string, restarts: number, size: RunnerSize): string {
40+ return `job-${jobId}${restarts ? `-r${restarts}` : ""}${size === "large" ? "" : `.${size}`}`;
41+}
42+
43+export function sizeOfContainer(name: string): RunnerSize {
44+ const m = /\.(medium|small|tiny)$/.exec(name);
45+ return (m?.[1] as RunnerSize | undefined) ?? "large";
46+}
47+
48+/** One earlier run of the same job: how it ended, on which size, and what it used. */
49+export type SizedRun = { conclusion: string | null; size: RunnerSize; usage: JobUsage | null };
50+
51+/** Headroom a smaller runner must leave over what the job used: memory and disk are sampled, so a peak can be missed. */
52+export const MEMORY_HEADROOM = { factor: 1.5, plusGiB: 0.5 };
53+export const DISK_HEADROOM = { factor: 1.25, plusGB: 1 };
54+/**
55+ * CPU headroom: a smaller runner must have this many times the cores the job kept busy at its busiest, so a
56+ * burst shorter than a sample (or a little busier than the last runs) still has room and doesn't queue.
57+ */
58+export const CPU_HEADROOM = 1.5;
59+/**
60+ * A run on a smaller runner that took longer than this over the job's typical time on large (the median of
61+ * its measured large runs) was slowed by it, or may have been: the job goes back to large.
62+ */
63+export const SLOWER = { factor: 1.2, plusSeconds: 10 };
64+/** A run busy at its runner's limit this share of its time may have wanted more: no smaller runner than that one. */
65+export const SATURATED = 0.05;
66+/** Earlier runs looked at, and how many measured ones a job needs before it can get a smaller runner. */
67+export const HISTORY = 5;
68+export const MIN_MEASURED = 3;
69+
70+/**
71+ * The most cores a run kept busy in any 2-second window, counted at its level's upper bound (never below
72+ * what was measured). A smaller runner must have at least this many vCPUs: its average says little about a
73+ * test suite's bursts, and a job that is short of CPU even for seconds can miss its own timeouts.
74+ */
75+export function peakCores(u: JobUsage): number {
76+ let top = 0;
77+ u.busySeconds.forEach((secs, i) => {
78+ if (secs > 0) top = Math.max(top, Math.min(BUSY_BUCKETS[i] ?? Infinity, u.cores));
79+ });
80+ return Math.max(top, u.coresMax);
81+}
82+
83+/** Why a size would not do for a run, or null when it would. */
84+function unfit(size: RunnerSize, r: SizedRun & { usage: JobUsage }): string | null {
85+ const s = RUNNER_SIZES[size];
86+ const u = r.usage;
87+ if (u.memoryPeak / 1024 ** 3 * MEMORY_HEADROOM.factor + MEMORY_HEADROOM.plusGiB > s.memoryGiB) return "memory";
88+ if (u.diskPeak / 1e9 * DISK_HEADROOM.factor + DISK_HEADROOM.plusGB > s.diskGB) return "disk";
89+ // Held back by its own runner's CPU: it may need more than it showed, so nothing smaller than that runner.
90+ if (u.saturated > SATURATED && s.vcpu <= RUNNER_SIZES[r.size].vcpu) return "cpu";
91+ if (peakCores(u) * CPU_HEADROOM > s.vcpu) return "cpu";
92+ return null;
93+}
94+
95+export type SizeChoice = { size: RunnerSize; why: string };
96+
97+/**
98+ * The size for a hosted job from its earlier runs (the same workflow and job name, newest first). It is
99+ * `large` unless each of its last few runs (at least MIN_MEASURED, all measured) fits a smaller one with headroom: peak memory
100+ * and disk well inside it, and half as many cores again as it kept busy at its busiest. A run that ran out
101+ * of memory, failed on a smaller runner, took longer there than the job takes on large, or wasn't measured
102+ * keeps the job on `large`, so a smaller runner is never what makes a job fail or slow down twice. A re-run
103+ * of a failed job always gets `large`. Runs past the last few only give the job's typical time on large.
104+ */
105+export function chooseSize(history: readonly SizedRun[], o: { rerun?: boolean } = {}): SizeChoice {
106+ if (o.rerun) return { size: "large", why: "a re-run runs on the large runner" };
107+ const runs = history.filter((r) => r.conclusion !== "cancelled" && r.conclusion !== "skipped").slice(0, HISTORY);
108+ if (!runs.length) return { size: "large", why: "no earlier run of this job to size it by" };
109+ for (const r of runs) {
110+ if (!r.usage) return { size: "large", why: "an earlier run of this job wasn't measured" };
111+ if (r.usage.oomKills) return { size: "large", why: "an earlier run of this job ran out of memory" };
112+ if (r.size !== "large" && r.conclusion !== "success") return { size: "large", why: `an earlier run of this job failed on a ${r.size} runner` };
113+ }
114+ const onLarge = history
115+ .filter((r) => r.size === "large" && r.conclusion === "success" && r.usage)
116+ .map((r) => r.usage!.seconds)
117+ .sort((a, b) => a - b);
118+ if (onLarge.length) {
119+ const typical = onLarge[Math.floor((onLarge.length - 1) / 2)]!;
120+ const slow = runs.find((r) => r.size !== "large" && r.usage!.seconds > typical * SLOWER.factor + SLOWER.plusSeconds);
121+ if (slow) return { size: "large", why: `an earlier run of this job took longer on a ${slow.size} runner than it does on large` };
122+ }
123+ if (runs.length < MIN_MEASURED) return { size: "large", why: `fewer than ${MIN_MEASURED} measured runs of this job to size it by` };
124+ const measured = runs as Array<SizedRun & { usage: JobUsage }>;
125+ for (const size of ASCENDING) {
126+ if (size === "large") break;
127+ if (measured.every((r) => !unfit(size, r))) {
128+ const peak = Math.max(...measured.map((r) => r.usage.memoryPeak)) / 1024 ** 3;
129+ return { size, why: `sized by this job's last ${measured.length} runs (peak memory ${peak.toFixed(1)} GiB)` };
130+ }
131+ }
132+ return { size: "large", why: "this job uses what a smaller runner doesn't have" };
133+}
134+
135+/** The log line naming the job's runner size and why it got it. */
136+export function describeSize(size: RunnerSize, why: string, pinned: boolean): string {
137+ const s = RUNNER_SIZES[size];
138+ const cores = s.vcpu >= 1 ? `${s.vcpu} vCPU` : `${s.vcpu} of a vCPU`;
139+ return `Runner size: ${size} (${cores}, ${s.memoryGiB} GiB of memory, ${s.diskGB} GB of disk), ${pinned ? "asked for by runs-on" : `${why}; runs-on: diffinery-large (or -medium, -small, -tiny) sets a size`}`;
140+}
141+
142+/**
143+ * A runner's usage report as stored (JSON), or null when it isn't one. The runner holding the job's token
144+ * writes it, so only known fields are kept, as finite numbers in range: at worst a job misreports its own
145+ * use and is sized for that, and a run that then fails on a smaller runner goes back to large.
146+ */
147+export function cleanUsage(raw: unknown): string | null {
148+ if (!raw || typeof raw !== "object") return null;
149+ const u = raw as Record<string, unknown>;
150+ const num = (k: string, max: number) => {
151+ const v = u[k];
152+ return typeof v === "number" && Number.isFinite(v) && v >= 0 ? Math.min(v, max) : null;
153+ };
154+ const out: Record<string, unknown> = {};
155+ const limits: Record<string, number> = { seconds: 1e6, cpuSeconds: 1e8, coresP95: 1024, coresMax: 1024, saturated: 1, memoryPeak: 2 ** 50, diskPeak: 2 ** 50, cores: 1024, memory: 2 ** 50, disk: 2 ** 50 };
156+ for (const [k, max] of Object.entries(limits)) {
157+ const v = num(k, max);
158+ if (v === null) return null;
159+ out[k] = v;
160+ }
161+ out.oomKills = u.oomKills === null ? null : num("oomKills", 1e6);
162+ const busy = u.busySeconds;
163+ if (!Array.isArray(busy) || busy.length !== BUSY_BUCKETS.length || !busy.every((v) => typeof v === "number" && Number.isFinite(v) && v >= 0)) return null;
164+ out.busySeconds = busy.map((v: number) => Math.min(v, 1e6));
165+ return JSON.stringify(out);
166+}
modifiedapps/diffinery/test/e2e/entry.ts+1−1View file
@@ -6,7 +6,7 @@
66
77 export { RepoDO } from "../../src/repo/RepoDO.js";
88 export { SchedulerDO } from "../../src/scheduler.js";
9-export { RunnerContainer, SandboxContainer } from "../../src/runner/containers.js";
9+export { RunnerContainer, RunnerContainerMedium, RunnerContainerSmall, RunnerContainerTiny, SandboxContainer } from "../../src/runner/containers.js";
1010 export { RunWorkflow } from "../../src/runner/workflow.js";
1111 export { ImportWorkflow } from "../../src/imports/workflow.js";
1212 export { AccountWorkflow } from "../../src/accounts/workflow.js";
addedapps/diffinery/test/unit/runner-sizes.test.ts+277−0View file
@@ -1,0 +1,277 @@
1+import { describe, expect, it } from "vitest";
2+import { DatabaseSync } from "node:sqlite";
3+import { BUSY_BUCKETS, type JobUsage } from "@diffinery/shared";
4+import { parseWorkflow } from "@diffinery/workflow-yaml";
5+import { Actions } from "../../src/repo/actions.js";
6+import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
7+import { chooseSize, cleanUsage, containerName, describeSize, sizeLabel, sizeOfContainer, peakCores, type SizedRun } from "../../src/runner/sizes.js";
8+import { testD1 } from "./stubs/d1.js";
9+import type { RepoDO } from "../../src/repo/RepoDO.js";
10+import type { Env } from "../../src/env.js";
11+import type { Caller } from "../../src/repo/types.js";
12+
13+const GiB = 1024 ** 3;
14+
15+/** A measured run: `busy` maps busy cores (a bucket's upper bound) to seconds spent there. */
16+function usage(o: { seconds?: number; memGiB?: number; diskGB?: number; busy?: Record<number, number>; cores?: number; saturated?: number; oom?: number | null } = {}): JobUsage {
17+ const seconds = o.seconds ?? 120;
18+ const busy = o.busy ?? { 0.1: seconds };
19+ const busySeconds = BUSY_BUCKETS.map((b) => busy[b] ?? 0);
20+ return {
21+ seconds,
22+ cpuSeconds: Object.entries(busy).reduce((a, [c, s]) => a + Number(c) * s, 0),
23+ coresP95: 0,
24+ coresMax: 0,
25+ saturated: o.saturated ?? 0,
26+ busySeconds,
27+ memoryPeak: (o.memGiB ?? 0.3) * GiB,
28+ diskPeak: (o.diskGB ?? 1.5) * 1e9,
29+ cores: o.cores ?? 4,
30+ memory: 12 * GiB,
31+ disk: 20e9,
32+ oomKills: o.oom === undefined ? 0 : o.oom,
33+ };
34+}
35+const ran = (u: JobUsage | null, size: SizedRun["size"] = "large", conclusion = "success"): SizedRun => ({ conclusion, size, usage: u });
36+/** The run, then enough measured runs like it behind it to size by. */
37+const runs = (...first: SizedRun[]) => [...first, ...Array.from({ length: Math.max(0, 3 - first.length) }, () => ran(usage()))];
38+
39+describe("runner sizes", () => {
40+ it("reads a size from runs-on labels, the largest when several", () => {
41+ expect(sizeLabel(["ubuntu-latest"])).toBeNull();
42+ expect(sizeLabel(["ubuntu-latest", "diffinery-small"])).toBe("small");
43+ expect(sizeLabel(["Diffinery-Tiny", "diffinery-medium"])).toBe("medium");
44+ expect(sizeLabel(["diffinery-huge"])).toBeNull();
45+ });
46+
47+ it("names containers by size, and large ones as before", () => {
48+ expect(containerName("j1", 0, "large")).toBe("job-j1");
49+ expect(containerName("j1", 2, "large")).toBe("job-j1-r2");
50+ expect(containerName("j1", 1, "small")).toBe("job-j1-r1.small");
51+ expect(sizeOfContainer("job-j1-r1.small")).toBe("small");
52+ expect(sizeOfContainer("job-j1")).toBe("large");
53+ });
54+
55+ it("stays large without history, after a re-run, or when a run wasn't measured", () => {
56+ expect(chooseSize([]).size).toBe("large");
57+ expect(chooseSize(runs(), { rerun: true }).size).toBe("large");
58+ expect(chooseSize(runs(ran(usage()), ran(null))).size).toBe("large");
59+ // Two measured runs aren't enough to size by.
60+ expect(chooseSize([ran(usage()), ran(usage())]).why).toContain("fewer than 3");
61+ });
62+
63+ it("gives a job that waits on the network the smallest runner", () => {
64+ const c = chooseSize(runs(ran(usage({ memGiB: 0.3 })), ran(usage({ memGiB: 0.25 }))));
65+ expect(c.size).toBe("tiny");
66+ expect(c.why).toContain("last 3 runs");
67+ });
68+
69+ it("keeps memory headroom: 1.5 times the peak plus half a GiB", () => {
70+ // 0.4 GiB * 1.5 + 0.5 = 1.1 GiB: more than tiny's 1 GiB, so small.
71+ expect(chooseSize(runs(ran(usage({ memGiB: 0.4 })))).size).toBe("small");
72+ // 5.5 GiB * 1.5 + 0.5 = 8.75 GiB: more than medium's 8 GiB, so large even with little CPU.
73+ expect(chooseSize(runs(ran(usage({ memGiB: 5.5 })))).size).toBe("large");
74+ // The largest of the recent runs counts.
75+ expect(chooseSize(runs(ran(usage({ memGiB: 0.2 })), ran(usage({ memGiB: 5.5 })))).size).toBe("large");
76+ });
77+
78+ it("keeps disk headroom", () => {
79+ // 2.5 GB * 1.25 + 1 = 4.1 GB: more than tiny's 4 GB.
80+ expect(chooseSize(runs(ran(usage({ diskGB: 2.5 })))).size).toBe("small");
81+ expect(chooseSize(runs(ran(usage({ diskGB: 14 })))).size).toBe("large");
82+ });
83+
84+ it("never gives a job fewer vCPUs than it kept busy, even for a few seconds", () => {
85+ // Two minutes at 3 cores: CPU-bound, so large.
86+ const busy = usage({ seconds: 130, busy: { 0.1: 10, 3: 120 }, memGiB: 2 });
87+ expect(peakCores(busy)).toBe(3);
88+ expect(chooseSize(runs(ran(busy))).size).toBe("large");
89+ // Mostly idle, but a few seconds at 3 cores (a test suite's burst): still large, since timing-sensitive
90+ // work can miss its own timeouts on fewer cores however short the burst.
91+ const burst = usage({ seconds: 300, busy: { 0.1: 290, 3: 10 }, memGiB: 2 });
92+ expect(chooseSize(runs(ran(burst))).size).toBe("large");
93+ // The peak reported by the runner counts even where the buckets show less, with half as much again
94+ // to spare: 0.4 cores needs 0.6, more than small's half vCPU.
95+ expect(chooseSize(runs(ran({ ...usage(), coresMax: 0.4 }))).size).toBe("medium");
96+ expect(chooseSize(runs(ran({ ...usage(), coresMax: 0.3 }))).size).toBe("small");
97+ // Steady single-core work fits medium's 2 vCPU but not half a vCPU.
98+ expect(chooseSize(runs(ran(usage({ seconds: 200, busy: { 1: 200 }, memGiB: 1 })))).size).toBe("medium");
99+ });
100+
101+ it("moves a job back to large after it ran out of memory or failed on a smaller runner", () => {
102+ expect(chooseSize(runs(ran(usage({ oom: 1 })))).size).toBe("large");
103+ expect(chooseSize(runs(ran(usage(), "tiny", "failure"))).size).toBe("large");
104+ expect(chooseSize(runs(ran(usage(), "small", "timed_out"))).size).toBe("large");
105+ // A failure on large isn't the size's doing; cancelled runs don't count.
106+ expect(chooseSize(runs(ran(usage(), "large", "failure"))).size).toBe("tiny");
107+ expect(chooseSize([ran(null, "large", "cancelled"), ...runs()]).size).toBe("tiny");
108+ });
109+
110+ it("moves a job back to large after it took longer on a smaller runner", () => {
111+ const large = (seconds: number) => ran(usage({ seconds }));
112+ const history = [large(100), large(110), large(90), large(300)];
113+ // Typical on large: 100 s (the median). 125 s on small is within 1.2x + 10 s; 131 s isn't.
114+ expect(chooseSize([ran(usage({ seconds: 125 }), "tiny"), ...history]).size).toBe("tiny");
115+ const c = chooseSize([ran(usage({ seconds: 131 }), "tiny"), ...history]);
116+ expect(c.size).toBe("large");
117+ expect(c.why).toContain("took longer on a tiny runner");
118+ // Large runs further back still give the typical time.
119+ const old = [...Array.from({ length: 5 }, () => ran(usage({ seconds: 300 }), "tiny")), large(100), large(100)];
120+ expect(chooseSize(old).size).toBe("large");
121+ });
122+
123+ it("doesn't go below a runner whose CPU held the job back", () => {
124+ // Busy at small's half vCPU most of the time: it may need more, so at least medium.
125+ const capped = usage({ seconds: 100, busy: { 0.5: 100 }, cores: 0.5, saturated: 0.9 });
126+ expect(chooseSize(runs(ran(capped, "small"))).size).toBe("medium");
127+ });
128+
129+ it("names the size and why in the log", () => {
130+ expect(describeSize("small", "sized by this job's last run (peak memory 0.4 GiB)", false)).toBe(
131+ "Runner size: small (0.5 of a vCPU, 4 GiB of memory, 8 GB of disk), sized by this job's last run (peak memory 0.4 GiB); runs-on: diffinery-large (or -medium, -small, -tiny) sets a size",
132+ );
133+ expect(describeSize("large", "", true)).toContain("asked for by runs-on");
134+ });
135+
136+ it("keeps only a well-formed usage report", () => {
137+ const u = usage();
138+ expect(JSON.parse(cleanUsage({ ...u, extra: "x" })!)).toEqual(u);
139+ expect(cleanUsage({ ...u, memoryPeak: -1 })).toBeNull();
140+ expect(cleanUsage({ ...u, busySeconds: [1, 2] })).toBeNull();
141+ expect(cleanUsage({ ...u, seconds: "1" })).toBeNull();
142+ expect(cleanUsage(null)).toBeNull();
143+ });
144+});
145+
146+/** The Durable Object's SqlStorage over node:sqlite. */
147+function sqlStorage() {
148+ const db = new DatabaseSync(":memory:");
149+ return {
150+ exec(query: string, ...args: unknown[]) {
151+ const params = args.map((a) => (a === undefined ? null : a)) as Array<string | number | null>;
152+ let rows: unknown[] = [];
153+ if (!params.length && query.trim().replace(/;\s*$/, "").includes(";")) db.exec(query);
154+ else rows = db.prepare(query).all(...params);
155+ return {
156+ toArray: () => rows as never[],
157+ one: () => {
158+ if (rows.length !== 1) throw new Error(`expected one row, got ${rows.length}`);
159+ return rows[0] as never;
160+ },
161+ };
162+ },
163+ };
164+}
165+
166+const ADA: Caller = { id: "u1", login: "ada", kind: "user", role: "write" };
167+
168+/** Actions over an in-memory repository, with stand-ins for each runner class recording what started where. */
169+function fixture(bindings: string[] = ["RUNNER", "RUNNER_MEDIUM", "RUNNER_SMALL", "RUNNER_TINY"], refuse: (binding: string) => boolean = () => false) {
170+ const sql = sqlStorage();
171+ sql.exec(REPO_SCHEMA);
172+ migrateRepoSchema(sql as unknown as SqlStorage);
173+ const DB = testD1();
174+ DB.sqlite.prepare("INSERT INTO accounts (id, login, kind, status, created_at) VALUES ('acme', 'acme', 'org', 'active', ?)").run(new Date().toISOString());
175+ const started: Array<{ binding: string; name: string; vcpu?: number }> = [];
176+ const stopped: Array<{ binding: string; name: string }> = [];
177+ const repo = {
178+ sql,
179+ meta: { repoId: "r", owner: "acme", ownerId: "acme", ownerKind: "org", name: "app", defaultBranch: "main" },
180+ headRef: "refs/heads/main",
181+ getRef: () => null,
182+ upsertCheck: () => {},
183+ broadcast: () => {},
184+ emit: () => {},
185+ armAlarm: () => {},
186+ deploys: { jobCompleted: () => {}, protection: () => null },
187+ runtime: { sweepJob: async () => {} },
188+ policy: { agentOrigin: () => false },
189+ } as unknown as RepoDO;
190+ const ns = (binding: string) => ({
191+ idFromName: (n: string) => n,
192+ get: (n: string) => ({
193+ startJob: async (input: { vcpu?: number }) => {
194+ if (refuse(binding)) throw new Error("there is no container instance that can be provided to this durable object");
195+ started.push({ binding, name: n, vcpu: input.vcpu });
196+ },
197+ stopJob: async () => void stopped.push({ binding, name: n }),
198+ }),
199+ });
200+ const env = {
201+ DB,
202+ PUBLIC_URL: "https://d.test",
203+ BUCKET: { put: async () => {}, get: async () => null },
204+ ...Object.fromEntries(bindings.map((b) => [b, ns(b)])),
205+ RUN_WORKFLOW: { create: async () => {}, get: async () => ({ sendEvent: async () => {} }) },
206+ } as unknown as Env;
207+ const actions = new Actions(repo, env);
208+ async function run(runsOn = "ubuntu-latest") {
209+ const yaml = `name: w\non: push\njobs:\n deploy:\n runs-on: ${runsOn}\n steps: [{ run: echo hi }]\n`;
210+ const { workflow } = parseWorkflow(yaml, ".github/workflows/w.yml");
211+ Object.defineProperty(actions, "loadWorkflows", { value: async () => [{ path: ".github/workflows/w.yml", workflow, errors: [] }], configurable: true });
212+ const r = (await actions.createRun(workflow!, { event: "push", ref: "refs/heads/main", sha: "abc", payload: {}, caller: ADA, title: "t" }))!;
213+ expect((await actions.plan(r.id)).error).toBeUndefined();
214+ const { jobIds } = await actions.startJobKey(r.id, "deploy");
215+ return jobIds[0]!;
216+ }
217+ const complete = (jobId: string, conclusion: "success" | "failure", u: JobUsage | null) =>
218+ (actions as unknown as { completeJob: (id: string, c: unknown) => Promise<void> }).completeJob(jobId, { conclusion, outputs: {}, ...(u ? { usage: u } : {}) });
219+ const job = (id: string) => sql.exec("SELECT * FROM jobs WHERE id = ?", id).one() as { container: string; size_note: string; usage: string | null; runner_size: string | null };
220+ return { actions, started, stopped, run, complete, job };
221+}
222+
223+describe("hosted jobs on sized runners", () => {
224+ it("runs a first job large, then sizes it by what it used", async () => {
225+ const f = fixture();
226+ const first = await f.run();
227+ expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER", name: `job-${first}`, vcpu: 4 });
228+ expect(f.job(first).size_note).toContain("no earlier run");
229+ await f.complete(first, "success", usage({ memGiB: 0.3 }));
230+ expect(JSON.parse(f.job(first).usage!).memoryPeak).toBe(0.3 * GiB);
231+ // Large until three measured runs say otherwise.
232+ for (let i = 0; i < 2; i++) {
233+ const id = await f.run();
234+ expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER" });
235+ await f.complete(id, "success", usage({ memGiB: 0.3 }));
236+ }
237+
238+ const second = await f.run();
239+ expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER_TINY", name: `job-${second}.tiny`, vcpu: 0.25 });
240+ expect(f.actions.getJob(second).runnerSize).toBe("tiny");
241+ expect(f.job(second).size_note).toMatch(/^Runner size: tiny/);
242+
243+ // It fails there: the next run is large again.
244+ await f.complete(second, "failure", usage({ memGiB: 0.3, cores: 0.25 }));
245+ const third = await f.run();
246+ expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER", name: `job-${third}` });
247+ expect(f.job(third).size_note).toContain("failed on a tiny runner");
248+ });
249+
250+ it("runs a job where runs-on asks, and stops it there", async () => {
251+ const f = fixture();
252+ const id = await f.run("[ubuntu-latest, diffinery-medium]");
253+ expect(f.job(id).runner_size).toBe("medium");
254+ expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER_MEDIUM", name: `job-${id}.medium`, vcpu: 2 });
255+ expect(f.job(id).size_note).toContain("asked for by runs-on");
256+ await (f.actions as unknown as { stopContainer: (j: unknown) => Promise<void> }).stopContainer(f.job(id));
257+ expect(f.stopped).toEqual([{ binding: "RUNNER_MEDIUM", name: `job-${id}.medium` }]);
258+ });
259+
260+ it("starts a job large when no smaller runner was free for it", async () => {
261+ const f = fixture(undefined, (b) => b === "RUNNER_TINY");
262+ for (let i = 0; i < 3; i++) await f.complete(await f.run(), "success", usage({ memGiB: 0.3 }));
263+ const second = await f.run();
264+ // The tiny runner had no machine: the job went back in line, and starts again on large.
265+ expect(f.job(second)).toMatchObject({ container: null });
266+ await f.actions.relaunchJob(second);
267+ expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER", name: `job-${second}` });
268+ expect(f.job(second).size_note).toContain("no smaller runner was free");
269+ });
270+
271+ it("runs every job large where the smaller runners aren't deployed", async () => {
272+ const f = fixture(["RUNNER"]);
273+ const id = await f.run("diffinery-small");
274+ expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER", name: `job-${id}`, vcpu: 4 });
275+ expect(f.job(id).size_note).toContain("no smaller runners");
276+ });
277+});
modifiedapps/diffinery/wrangler.jsonc+16−1View file
@@ -59,6 +59,10 @@
5959 { "name": "REPO", "class_name": "RepoDO" },
6060 { "name": "SCHEDULER", "class_name": "SchedulerDO" },
6161 { "name": "RUNNER", "class_name": "RunnerContainer" },
62+ // Smaller hosted runners (src/runner/sizes.ts), the same runner on smaller instance types.
63+ { "name": "RUNNER_MEDIUM", "class_name": "RunnerContainerMedium" },
64+ { "name": "RUNNER_SMALL", "class_name": "RunnerContainerSmall" },
65+ { "name": "RUNNER_TINY", "class_name": "RunnerContainerTiny" },
6266 { "name": "SANDBOX", "class_name": "SandboxContainer" },
6367 { "name": "IMPORTER", "class_name": "ImportContainer" },
6468 { "name": "PROJECT", "class_name": "ProjectDO" }
@@ -70,7 +74,9 @@
7074 // Agent tasks, one Durable Object per repository (agentic/ProjectDO.ts).
7175 { "tag": "v3", "new_sqlite_classes": ["ProjectDO"] },
7276 // Workflow schedules (scheduler.ts).
73- { "tag": "v4", "new_sqlite_classes": ["SchedulerDO"] }
77+ { "tag": "v4", "new_sqlite_classes": ["SchedulerDO"] },
78+ // Smaller hosted runner sizes (src/runner/sizes.ts).
79+ { "tag": "v5", "new_sqlite_classes": ["RunnerContainerMedium", "RunnerContainerSmall", "RunnerContainerTiny"] }
7480 ],
7581 "workflows": [
7682 { "name": "diffinery-run", "binding": "RUN_WORKFLOW", "class_name": "RunWorkflow" },
@@ -94,6 +100,12 @@
94100 // standard-4 (4 vCPU, 12 GiB, 20 GB): standard-2 has 1 vCPU, on which typecheck and test steps took
95101 // 2.3 to 2.6 times as long. Active CPU is what's billed, so a faster job costs about the same.
96102 { "class_name": "RunnerContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-4", "max_instances": 100, "rollout_active_grace_period": 23400, "rollout_step_percentage": 100 },
103+ // Smaller sizes for jobs that use little of a large runner (src/runner/sizes.ts): Cloudflare bills a container's
104+ // memory and disk as provisioned, used or not, so a job that waits on the network costs a large runner's memory.
105+ // A job gets one when runs-on asks for it, or when its own earlier runs show it fits with room to spare.
106+ { "class_name": "RunnerContainerMedium", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-3", "max_instances": 100, "rollout_active_grace_period": 23400, "rollout_step_percentage": 100 },
107+ { "class_name": "RunnerContainerSmall", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-1", "max_instances": 100, "rollout_active_grace_period": 23400, "rollout_step_percentage": 100 },
108+ { "class_name": "RunnerContainerTiny", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "basic", "max_instances": 100, "rollout_active_grace_period": 23400, "rollout_step_percentage": 100 },
97109 { "class_name": "SandboxContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-1", "max_instances": 50, "rollout_active_grace_period": 3600, "rollout_step_percentage": 100 },
98110 // GitHub imports: a mirror clone of a large repository needs disk and memory.
99111 { "class_name": "ImportContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-4", "max_instances": 20, "rollout_active_grace_period": 25200, "rollout_step_percentage": 100 }
@@ -136,6 +148,9 @@
136148 { "name": "REPO", "class_name": "RepoDO" },
137149 { "name": "SCHEDULER", "class_name": "SchedulerDO" },
138150 { "name": "RUNNER", "class_name": "RunnerContainer" },
151+ { "name": "RUNNER_MEDIUM", "class_name": "RunnerContainerMedium" },
152+ { "name": "RUNNER_SMALL", "class_name": "RunnerContainerSmall" },
153+ { "name": "RUNNER_TINY", "class_name": "RunnerContainerTiny" },
139154 // Sandboxes run in an `agent.mjs --sandbox-pool` process (SANDBOX_POOL_URL), not containers.
140155 { "name": "SANDBOX", "class_name": "SandboxPoolDO" },
141156 { "name": "IMPORTER", "class_name": "ImportContainer" },
modifiedcontainers/runner/src/agent.ts+13−0View file
@@ -1,6 +1,7 @@
11 import type { JobCompletion, JobSpec } from "@diffinery/shared";
22 import { LogBatcher, RunnerApi } from "./api.js";
33 import { JobRunner, errMsg } from "./job.js";
4+import { UsageSampler, describeUsage } from "./usage.js";
45
56 export interface AgentOptions {
67 /** `${apiUrl}/runner/jobs/${jobId}`, or null for offline mode (logs to stdout, no reports). */
@@ -20,6 +21,11 @@
2021 onRunner?: (r: JobRunner) => void;
2122 logIntervalMs?: number;
2223 baseEnv?: NodeJS.ProcessEnv;
24+ /**
25+ * Measure what the job uses of the machine and report it with the result (hosted runners, where the
26+ * container is the job's own machine; on a shared self-hosted machine the numbers would be the host's).
27+ */
28+ measureUsage?: boolean | UsageSampler;
2329 }
2430
2531 export function apiBaseFor(apiUrl: string | undefined | null, jobId: string): string | null {
@@ -56,12 +62,19 @@
5662 process.on("SIGTERM", sigterm);
5763 process.on("SIGINT", sigint);
5864 }
65+ const sampler = o.measureUsage instanceof UsageSampler ? o.measureUsage : o.measureUsage ? new UsageSampler() : null;
66+ sampler?.start();
5967 let completion: JobCompletion;
6068 try {
6169 completion = await runner.run();
6270 } catch (e) {
6371 completion = { conclusion: "failure", outputs: {}, failure: { message: `Runner error: ${errMsg(e)}` } };
6472 }
73+ const usage = sampler?.stop();
74+ if (usage) {
75+ completion.usage = usage;
76+ runner.logger.line(-1, describeUsage(usage));
77+ }
6578 if (batcher) {
6679 // After a SIGTERM the platform kills the container soon, so the result must not wait long on logs.
6780 const ok = await batcher.close(draining ? 30_000 : 120_000);
modifiedcontainers/runner/src/job.ts+1−0View file
@@ -322,6 +322,7 @@
322322 this.logger.line(-1, `Diffinery runner: job ${spec.jobKey} (${spec.name}) for ${repoFull}@${spec.sha.slice(0, 12)}`);
323323 const where = runnerInfo();
324324 if (Object.keys(where).length) this.logger.line(-1, `Runner: ${[where.image && `image ${where.image}`, where.location && `in ${where.location}`, where.instance && `container ${where.instance}`].filter(Boolean).join(", ")}`);
325+ if (spec.runnerSize) this.logger.line(-1, spec.runnerSize);
325326 this.logger.line(-1, `Workspace: ${this.workspace}`);
326327 // job-level env
327328 const ctx = this.contexts({ steps: {}, inputs: spec.inputs ?? {}, failed: false, inheritedEnv: {}, composite: false, depth: 0 }, {});
modifiedcontainers/runner/src/main.ts+3−0View file
@@ -16,6 +16,7 @@
1616 */
1717 import { readFile } from "node:fs/promises";
1818 import type { JobSpec } from "@diffinery/shared";
19+import { UsageSampler } from "./usage.js";
1920 import { apiBaseFor, runAgentJob } from "./agent.js";
2021 import { trustEgressCa } from "./egress.js";
2122 import { RunnerApi } from "./api.js";
@@ -148,6 +149,8 @@
148149 // Cloudflare gives a container 15 minutes after SIGTERM during a rollout; the last
149150 // two are for cancelling the steps and reporting the result before SIGKILL.
150151 drainOnSigtermMs: args.spec ? undefined : 13 * 60_000,
152+ // A hosted runner is its own machine (Cloudflare sets the Durable Object id in its container).
153+ measureUsage: !!env.CLOUDFLARE_DURABLE_OBJECT_ID || env.DIFFINERY_MEASURE_USAGE === "1" ? new UsageSampler({ cores: Number(env.DIFFINERY_RUNNER_VCPU) || undefined }) : false,
151154 });
152155 process.stdout.write(`[diffinery-runner] job finished: ${completion.conclusion}\n`);
153156 process.exit(completion.conclusion === "success" ? 0 : 1);
addedcontainers/runner/src/usage.ts+173−0View file
@@ -1,0 +1,173 @@
1+import { readFileSync, statfsSync } from "node:fs";
2+import os from "node:os";
3+import path from "node:path";
4+import { BUSY_BUCKETS, type JobUsage } from "@diffinery/shared";
5+
6+/** Reads of the machine at one moment: CPU time (busy and all, in ticks), memory and disk in use. */
7+type Reading = { busy: number; total: number; memUsed: number; memTotal: number; diskUsed: number; diskTotal: number; oom: number | null };
8+
9+export interface UsageSamplerOptions {
10+ /** Where /proc is (tests point it at a fake one). */
11+ procRoot?: string;
12+ /** The filesystem whose use is measured (the root filesystem holds the image, the workspace and the tool cache). */
13+ diskPath?: string;
14+ /** How often memory and disk are read; CPU is read on the same tick. */
15+ sampleMs?: number;
16+ /** Busy cores are measured over windows this long. */
17+ windowMs?: number;
18+ /** The machine's share of CPU in vCPUs (a quarter or half for the smallest hosted runners); by default the CPUs it sees. */
19+ cores?: number;
20+ /** CPUs the machine shows in /proc/stat (busy time is a share of all of them). */
21+ visible?: number;
22+}
23+
24+/**
25+ * Measures what a job uses of its machine while it runs: busy cores (from /proc/stat), memory in use
26+ * (MemTotal less MemAvailable from /proc/meminfo, which leaves out cache the kernel can drop), disk in
27+ * use, and out-of-memory kills (/proc/vmstat). A hosted container is its own virtual machine, so these
28+ * are the job's own numbers. Diffinery sizes the job's later runs by them (apps/diffinery/src/runner/sizes.ts).
29+ * Off Linux, or where /proc can't be read, it measures nothing and reports null.
30+ */
31+export class UsageSampler {
32+ private readonly proc: string;
33+ private readonly diskPath: string;
34+ private readonly sampleMs: number;
35+ private readonly windowMs: number;
36+ private readonly cores: number;
37+ private readonly visible: number;
38+ private timer: NodeJS.Timeout | null = null;
39+ private first: Reading | null = null;
40+ private windowStart: Reading | null = null;
41+ private windowStartedAt = 0;
42+ private startedAt = 0;
43+ private busyWindows: number[] = [];
44+ private busySeconds = BUSY_BUCKETS.map(() => 0);
45+ private cpuSeconds = 0;
46+ private memPeak = 0;
47+ private diskPeak = 0;
48+ private last: Reading | null = null;
49+
50+ constructor(o: UsageSamplerOptions = {}) {
51+ this.proc = o.procRoot ?? "/proc";
52+ this.diskPath = o.diskPath ?? "/";
53+ this.sampleMs = o.sampleMs ?? 500;
54+ this.windowMs = o.windowMs ?? 2000;
55+ this.visible = o.visible ?? os.availableParallelism();
56+ this.cores = o.cores && o.cores > 0 ? o.cores : this.visible;
57+ }
58+
59+ start(): void {
60+ const r = this.read();
61+ if (!r) return;
62+ this.first = this.windowStart = this.last = r;
63+ this.startedAt = this.windowStartedAt = Date.now();
64+ this.note(r);
65+ this.timer = setInterval(() => this.tick(), this.sampleMs);
66+ this.timer.unref();
67+ }
68+
69+ /** Stops sampling and returns what the job used, or null if nothing could be measured. */
70+ stop(): JobUsage | null {
71+ if (this.timer) clearInterval(this.timer);
72+ this.timer = null;
73+ if (!this.first) return null;
74+ this.tick(true);
75+ const first = this.first;
76+ const last = this.last ?? first;
77+ this.first = null;
78+ const seconds = (Date.now() - this.startedAt) / 1000;
79+ const sorted = [...this.busyWindows].sort((a, b) => a - b);
80+ const pct = (p: number) => (sorted.length ? sorted[Math.min(sorted.length - 1, Math.floor(p * sorted.length))]! : 0);
81+ const round = (n: number, d = 2) => Math.round(n * 10 ** d) / 10 ** d;
82+ return {
83+ seconds: round(seconds, 1),
84+ cpuSeconds: round(this.cpuSeconds, 1),
85+ coresP95: round(pct(0.95)),
86+ coresMax: round(sorted.length ? sorted[sorted.length - 1]! : 0),
87+ saturated: round(sorted.length ? sorted.filter((b) => b >= saturatedAt(this.cores)).length / sorted.length : 0, 3),
88+ busySeconds: this.busySeconds.map((s) => round(s, 1)),
89+ memoryPeak: this.memPeak,
90+ diskPeak: this.diskPeak,
91+ cores: this.cores,
92+ memory: last.memTotal,
93+ disk: last.diskTotal,
94+ oomKills: first.oom !== null && last.oom !== null ? Math.max(0, last.oom - first.oom) : null,
95+ };
96+ }
97+
98+ private tick(final = false): void {
99+ const r = this.read();
100+ if (!r) return;
101+ this.last = r;
102+ this.note(r);
103+ const now = Date.now();
104+ if (this.windowStart && (final || now - this.windowStartedAt >= this.windowMs)) {
105+ const dTotal = r.total - this.windowStart.total;
106+ const dBusy = r.busy - this.windowStart.busy;
107+ const secs = (now - this.windowStartedAt) / 1000;
108+ if (dTotal > 0 && secs > 0) {
109+ const busyCores = Math.min(this.cores, Math.max(0, (dBusy / dTotal) * this.visible));
110+ // A last window much shorter than the rest says little about how busy the job was.
111+ if (!final || secs >= this.windowMs / 2000) this.busyWindows.push(busyCores);
112+ this.cpuSeconds += busyCores * secs;
113+ const i = BUSY_BUCKETS.findIndex((b) => busyCores <= b);
114+ this.busySeconds[i < 0 ? BUSY_BUCKETS.length - 1 : i]! += secs;
115+ }
116+ this.windowStart = r;
117+ this.windowStartedAt = now;
118+ }
119+ }
120+
121+ private note(r: Reading): void {
122+ this.memPeak = Math.max(this.memPeak, r.memUsed);
123+ this.diskPeak = Math.max(this.diskPeak, r.diskUsed);
124+ }
125+
126+ private read(): Reading | null {
127+ try {
128+ const stat = readFileSync(path.join(this.proc, "stat"), "utf8");
129+ const cpu = stat.split("\n", 1)[0]!.trim().split(/\s+/);
130+ if (cpu[0] !== "cpu") return null;
131+ // user nice system idle iowait irq softirq steal: busy is what ran here, not idle, waiting or stolen time.
132+ const [user, nice, system, idle, iowait, irq, softirq, steal] = cpu.slice(1, 9).map((v) => Number(v) || 0);
133+ const busy = user! + nice! + system! + irq! + softirq!;
134+ const total = busy + idle! + iowait! + steal!;
135+ const mem = readFileSync(path.join(this.proc, "meminfo"), "utf8");
136+ const kb = (name: string) => Number(new RegExp(`^${name}:\\s+(\\d+)`, "m").exec(mem)?.[1] ?? NaN) * 1024;
137+ const memTotal = kb("MemTotal");
138+ const memAvailable = kb("MemAvailable");
139+ if (!Number.isFinite(memTotal) || !Number.isFinite(memAvailable)) return null;
140+ let diskUsed = 0;
141+ let diskTotal = 0;
142+ try {
143+ const fs = statfsSync(this.diskPath);
144+ diskTotal = fs.blocks * fs.bsize;
145+ diskUsed = (fs.blocks - fs.bfree) * fs.bsize;
146+ } catch {
147+ /* no disk numbers */
148+ }
149+ let oom: number | null = null;
150+ try {
151+ const m = /^oom_kill\s+(\d+)/m.exec(readFileSync(path.join(this.proc, "vmstat"), "utf8"));
152+ if (m) oom = Number(m[1]);
153+ } catch {
154+ /* an older kernel doesn't count them */
155+ }
156+ return { busy, total, memUsed: memTotal - memAvailable, memTotal, diskUsed, diskTotal, oom };
157+ } catch {
158+ return null;
159+ }
160+ }
161+}
162+
163+/** Busy cores at which a machine of `cores` vCPUs counts as held back by its CPU. */
164+export const saturatedAt = (cores: number) => Math.max(cores * 0.9, cores - 0.25);
165+
166+const gib = (b: number) => `${(b / 1024 ** 3).toFixed(1)} GiB`;
167+const gb = (b: number) => `${(b / 1e9).toFixed(1)} GB`;
168+
169+/** One log line saying what the job used of its machine. */
170+export function describeUsage(u: JobUsage): string {
171+ const avg = u.seconds > 0 ? u.cpuSeconds / u.seconds : 0;
172+ return `Resources: ${avg.toFixed(1)} of ${u.cores} vCPUs busy on average (${u.coresP95.toFixed(1)} at the 95th percentile), peak memory ${gib(u.memoryPeak)} of ${gib(u.memory)}, peak disk ${gb(u.diskPeak)} of ${gb(u.disk)}${u.oomKills ? `, ${u.oomKills} process(es) killed for want of memory` : ""}`;
173+}
addedcontainers/runner/test/usage.test.ts+81−0View file
@@ -1,0 +1,81 @@
1+import { mkdirSync, writeFileSync } from "node:fs";
2+import path from "node:path";
3+import { describe, expect, it, vi } from "vitest";
4+import { BUSY_BUCKETS } from "@diffinery/shared";
5+import { UsageSampler, describeUsage, saturatedAt } from "../src/usage.js";
6+import { tmp } from "./helpers.js";
7+
8+/** A fake /proc: CPU ticks (busy, idle) summed over all CPUs, memory in use and an OOM kill count. */
9+function fakeProc(dir: string) {
10+ mkdirSync(dir, { recursive: true });
11+ return (busy: number, idle: number, usedMiB: number, oom = 0, totalMiB = 4096) => {
12+ // user nice system idle iowait irq softirq steal: busy split over user and system; steal isn't busy.
13+ writeFileSync(path.join(dir, "stat"), `cpu ${busy - 10} 0 10 ${idle} 0 0 0 7 0 0\ncpu0 1 2 3\n`);
14+ writeFileSync(path.join(dir, "meminfo"), `MemTotal: ${totalMiB * 1024} kB\nMemFree: 1 kB\nMemAvailable: ${(totalMiB - usedMiB) * 1024} kB\n`);
15+ writeFileSync(path.join(dir, "vmstat"), `pgfault 5\noom_kill ${oom}\n`);
16+ };
17+}
18+
19+describe("usage sampler", () => {
20+ it("measures busy cores per window, peak memory and OOM kills", () => {
21+ vi.useFakeTimers({ now: 0 });
22+ try {
23+ const proc = path.join(tmp(), "proc");
24+ const set = fakeProc(proc);
25+ set(0, 0, 500);
26+ const s = new UsageSampler({ procRoot: proc, diskPath: proc, sampleMs: 500, windowMs: 2000, visible: 4 });
27+ s.start();
28+ // First 2 s: 1 of 4 CPUs busy (busy is a quarter of all ticks). Memory peaks at 1500 MiB mid-window.
29+ set(100, 300, 1500);
30+ vi.advanceTimersByTime(1000);
31+ set(200, 600, 800);
32+ vi.advanceTimersByTime(1000);
33+ // Next 2 s: all 4 busy, and one OOM kill.
34+ set(1000, 600, 900, 1);
35+ vi.advanceTimersByTime(2000);
36+ const u = s.stop()!;
37+ expect(u.seconds).toBe(4);
38+ expect(u.cores).toBe(4);
39+ expect(u.cpuSeconds).toBeCloseTo(10, 1);
40+ expect(u.coresMax).toBe(4);
41+ expect(u.memoryPeak).toBe(1500 * 1024 * 1024);
42+ expect(u.memory).toBe(4096 * 1024 * 1024);
43+ expect(u.oomKills).toBe(1);
44+ expect(u.saturated).toBe(0.5);
45+ expect(u.busySeconds).toHaveLength(BUSY_BUCKETS.length);
46+ expect(u.busySeconds[BUSY_BUCKETS.indexOf(1)]).toBe(2);
47+ expect(u.busySeconds[BUSY_BUCKETS.indexOf(4)]).toBe(2);
48+ expect(describeUsage(u)).toBe("Resources: 2.5 of 4 vCPUs busy on average (4.0 at the 95th percentile), peak memory 1.5 GiB of 4.0 GiB, peak disk " + `${(u.diskPeak / 1e9).toFixed(1)} GB of ${(u.disk / 1e9).toFixed(1)} GB, 1 process(es) killed for want of memory`);
49+ } finally {
50+ vi.useRealTimers();
51+ }
52+ });
53+
54+ it("counts busy cores against a fractional share of CPU", () => {
55+ vi.useFakeTimers({ now: 0 });
56+ try {
57+ const proc = path.join(tmp(), "proc");
58+ const set = fakeProc(proc);
59+ set(0, 0, 100);
60+ // One CPU visible, a quarter of it allotted: a quarter of the ticks busy is all it has.
61+ const s = new UsageSampler({ procRoot: proc, diskPath: proc, windowMs: 2000, visible: 1, cores: 0.25 });
62+ s.start();
63+ set(50, 150, 100);
64+ vi.advanceTimersByTime(2000);
65+ const u = s.stop()!;
66+ expect(u.cores).toBe(0.25);
67+ expect(u.coresMax).toBe(0.25);
68+ expect(u.saturated).toBe(1);
69+ expect(saturatedAt(0.25)).toBeCloseTo(0.225);
70+ expect(saturatedAt(4)).toBe(3.75);
71+ } finally {
72+ vi.useRealTimers();
73+ }
74+ });
75+
76+ it("measures nothing where /proc can't be read", () => {
77+ const s = new UsageSampler({ procRoot: path.join(tmp(), "missing") });
78+ s.start();
79+ expect(s.stop()).toBeNull();
80+ });
81+});
modifiedpackages/shared/src/runner.ts+34−0View file
@@ -47,6 +47,8 @@
4747 }
4848
4949 export type JobSpec = {
50+ /** Hosted runners: a log line naming the runner's size and why the job got it (apps/diffinery/src/runner/sizes.ts). */
51+ runnerSize?: string;
5052 runId: string;
5153 jobId: string;
5254 /** Key of the job in the workflow file, e.g. `build`. */
@@ -128,8 +130,40 @@
128130 * and the job didn't succeed, so Diffinery may start it again on a new runner.
129131 */
130132 replaced?: boolean;
133+ /** What the job used of its machine, measured by the runner (hosted runners size later runs by it). */
134+ usage?: JobUsage;
131135 };
132136
137+/**
138+ * A job's use of its machine, sampled by the runner while the job ran (containers/runner/src/usage.ts).
139+ * Busy cores come from /proc/stat, memory from /proc/meminfo (total less available), disk from the
140+ * root filesystem. Sampled every half second, so a spike shorter than that can be missed.
141+ */
142+/** Upper bounds (busy cores) of the levels JobUsage.busySeconds counts. */
143+export const BUSY_BUCKETS: readonly number[] = [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 4, 6, 8, 16, 32, 64];
144+
145+export type JobUsage = {
146+ /** Seconds measured, and CPU seconds used (all cores) in that time. */
147+ seconds: number;
148+ cpuSeconds: number;
149+ /** Busy cores at the 95th percentile and at the most busy sample (2 s windows). */
150+ coresP95: number;
151+ coresMax: number;
152+ /** Share of samples (0 to 1) in which the job kept nearly all its cores busy (90%, or all but a quarter core): it was held back by the CPU. */
153+ saturated: number;
154+ /** Seconds spent at each level of busy cores: entry i counts windows with at most BUSY_BUCKETS[i] cores busy (and more than the one before). */
155+ busySeconds: number[];
156+ /** Peak memory and disk in use, in bytes. */
157+ memoryPeak: number;
158+ diskPeak: number;
159+ /** What the machine had: vCPUs (its share, which can be a fraction), memory and disk in bytes. */
160+ cores: number;
161+ memory: number;
162+ disk: number;
163+ /** Processes the kernel killed for want of memory while the job ran (null when it doesn't say). */
164+ oomKills: number | null;
165+};
166+
133167 /** Runner API paths, relative to `${apiUrl}/runner/jobs/${jobId}`. */
134168 export const RunnerPaths = {
135169 spec: "",