Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Runner sizes: measure busy cores every quarter second, and judge slowdowns by the job's own noise (#237)

* Runner sizes: measure busy cores every quarter second, and judge slowdowns by the job's own noise
test-bot authored 1 parent 92a4795commit 0c0b2f093457ba787be041cf7e2becde1f31e398Browse files
6 files changed+109−23
Files (6)
modifiedapps/diffinery/docs/actions/runners.md+3−3View file
@@ -88,9 +88,9 @@
8888
8989 How a size is picked for a job without a label:
9090
91-- The runner measures every hosted job: busy cores every 2 seconds, memory and disk in use every half second, and processes the kernel killed for want of memory. The last lines of the job's log say what it used, for example `Resources: 0.1 of 4 vCPUs busy on average (0.3 at the 95th percentile), peak memory 0.3 GiB of 12.0 GiB, peak disk 1.6 GB of 19.3 GB`. The job API (`GET /repos/{owner}/{repo}/actions/jobs/{job_id}`) returns it as `usage`, and the size as `runner_size`.
92-- The next run of the same job (the same workflow file and job name, so each matrix leg on its own) looks at up to its last 5 runs, and needs at least 3. It gets a smaller size only if every one of them would have fit it with room to spare: peak memory times 1.5, plus 0.5 GiB, and peak disk times 1.25, plus 1 GB, within the size; and at least half as many vCPUs again as the job kept busy in its busiest 2-second window, so a job that bursts onto several cores even briefly, such as a test suite, keeps a runner with room for that. Busy cores count as on the slowest host processor runners land on: Cloudflare's hosts mix AMD EPYC generations whose cores differ about 2x in speed, so a job measured on a fast host gets cores to spare for a slow one (a run whose host processor isn't known counts as on the fastest). A job that kept its runner's CPU busy to the limit never gets a smaller size than that one.
93-- A job stays on large when it has no measured runs yet, when one of those runs ran out of memory, when one failed or timed out on a smaller size, or when one took longer on a smaller size than the job typically takes on large (its median on the same host processor where there is one, plus a fifth and 10 seconds). After a smaller size ran a job out of memory, failed it or slowed it, the job stays on large for its next 20 runs, so it doesn't move back and forth. A re-run always runs on large, and so does a job that found no smaller runner free when it first started. So a smaller size is never the reason a job fails or slows down twice.
91+- The runner measures every hosted job: busy cores, memory and disk in use every quarter second, and processes the kernel killed for want of memory. The last lines of the job's log say what it used, for example `Resources: 0.1 of 4 vCPUs busy on average (0.3 at the 95th percentile), peak memory 0.3 GiB of 12.0 GiB, peak disk 1.6 GB of 19.3 GB`. The job API (`GET /repos/{owner}/{repo}/actions/jobs/{job_id}`) returns it as `usage`, and the size as `runner_size`.
92+- The next run of the same job (the same workflow file and job name, so each matrix leg on its own) looks at up to its last 5 runs, and needs at least 3. It gets a smaller size only if every one of them would have fit it with room to spare: peak memory times 1.5, plus 0.5 GiB, and peak disk times 1.25, plus 1 GB, within the size; and at least half as many vCPUs again as the job kept busy in its busiest quarter second, so a job that bursts onto several cores even briefly, such as a test suite, keeps a runner with room for that. Busy cores count as on the slowest host processor runners land on: Cloudflare's hosts mix AMD EPYC generations whose cores differ about 2x in speed, so a job measured on a fast host gets cores to spare for a slow one (a run whose host processor isn't known counts as on the fastest). A job that kept its runner's CPU busy to the limit never gets a smaller size than that one.
93+- A job stays on large when it has no measured runs yet, when one of those runs ran out of memory, when one failed or timed out on a smaller size, or when one took longer on a smaller size than the job takes on large allowing for its own run-to-run noise there: more than a tenth plus 2 seconds over the median of its large runs (on the same host processor where there is one), and slower than all but one of its last ten large runs. A job whose earlier runs were measured before quarter-second measuring (October 16, 2026) is sized again from new runs. After a smaller size ran a job out of memory, failed it or slowed it, the job stays on large for its next 20 runs, so it doesn't move back and forth. A re-run always runs on large, and so does a job that found no smaller runner free when it first started. So a smaller size is never the reason a job fails or slows down twice.
9494 - The job's log names the size it got and why, near the top: `Runner size: tiny (0.25 of a vCPU, 1 GiB of memory, 4 GB of disk), sized by this job's last 5 runs (peak memory 0.3 GiB)`.
9595
9696 Set `runs-on: diffinery-large` to keep a job on large whatever it uses.
modifiedapps/diffinery/src/runner/sizes.ts+33−13View file
@@ -84,11 +84,19 @@
8484 */
8585 export const CPU_HEADROOM = 1.5;
8686 /**
87- * A run on a smaller runner that took longer than this over the job's typical time on large (the median of
88- * its measured large runs, on the same host processor when there are any) was slowed by it, or may have
89- * been: the job goes back to large.
87+ * A run on a smaller runner was slowed by it, or may have been, when it took longer than the job takes on
88+ * large plus its own run-to-run noise there: over a tenth and 2 seconds above the median of its measured
89+ * large runs (on the same host processor when there are any), and above the slowest but one of the last
90+ * ten of them (see slowLimit). The job then goes back to large.
9091 */
91-export const SLOWER = { factor: 1.2, plusSeconds: 10 };
92+export const SLOWER = { factor: 1.1, plusSeconds: 2 };
93+/**
94+ * Busy cores are measured over windows of at most this many seconds (the runner's own sampling). Shorter
95+ * windows catch a short burst at its height: averaged over 2 seconds, a third of a second on one core reads
96+ * as a sixth of a core, and a runner sized by that would run each such burst at half speed or less.
97+ * Runs measured with longer windows (runners from before 2026-10-16) count as not measured.
98+ */
99+export const MAX_WINDOW_SECONDS = 0.25;
92100 /** A run busy at its runner's limit this share of its time may have wanted more: no smaller runner than that one. */
93101 export const SATURATED = 0.05;
94102 /** Earlier runs looked at, and how many measured ones a job needs before it can get a smaller runner. */
@@ -101,7 +109,7 @@
101109 export const LOOKBACK = 20;
102110
103111 /**
104- * The most cores a run kept busy in any 2-second window, counted at its level's upper bound (never below
112+ * The most cores a run kept busy in any quarter-second window, counted at its level's upper bound (never below
105113 * what was measured). A smaller runner must have at least this many vCPUs: its average says little about a
106114 * test suite's bursts, and a job that is short of CPU even for seconds can miss its own timeouts.
107115 */
@@ -142,14 +150,17 @@
142150 const counted = history.filter((r) => r.conclusion !== "cancelled" && r.conclusion !== "skipped").slice(0, LOOKBACK);
143151 const runs = counted.slice(0, HISTORY);
144152 if (!runs.length) return { size: "large", why: "no earlier run of this job to size it by" };
145- for (const r of runs) if (!r.usage) return { size: "large", why: "an earlier run of this job wasn't measured" };
153+ for (const r of runs) {
154+ if (!r.usage) return { size: "large", why: "an earlier run of this job wasn't measured" };
155+ if (!(r.usage.windowSeconds! <= MAX_WINDOW_SECONDS)) return { size: "large", why: "an earlier run of this job wasn't measured closely enough (busy cores every quarter second)" };
156+ }
146157 const onLarge = (o.large ?? history).filter((r) => r.size === "large" && r.conclusion === "success" && r.usage);
147158 for (const r of counted) {
148159 if (r.usage?.oomKills) return { size: "large", why: "an earlier run of this job ran out of memory" };
149160 if (r.size === "large") continue;
150161 if (r.conclusion !== "success") return { size: "large", why: `an earlier run of this job failed on a ${r.size} runner` };
151- const typical = typicalSeconds(onLarge, r.cpu);
152- if (r.usage && typical !== null && r.usage.seconds > typical * SLOWER.factor + SLOWER.plusSeconds) {
162+ const limit = slowLimit(onLarge, r.cpu);
163+ if (r.usage && limit !== null && r.usage.seconds > limit) {
153164 return { size: "large", why: `an earlier run of this job took longer on a ${r.size} runner than it does on large` };
154165 }
155166 }
@@ -166,14 +177,20 @@
166177 }
167178
168179 /**
169- * A job's typical time on large: the median of its measured large runs on the same host processor as `cpu`
170- * (cores differ about 2x between hosts), or of all of them when none ran on that one; null with none.
180+ * The longest a run on a smaller runner may take before it counts as slowed: the job's time on large plus its
181+ * own noise there. Its measured large runs on the same host processor as `cpu` are compared (cores differ
182+ * about 2x between hosts), or all of them when none ran on that one; null with none. The limit is a tenth and
183+ * 2 seconds over their median, or, with 5 or more of them, their slowest but one when that is higher: a job
184+ * whose own large runs vary that much can't be told to be slower by less.
171185 */
172-function typicalSeconds(onLarge: readonly SizedRun[], cpu: string | null | undefined): number | null {
186+export function slowLimit(onLarge: readonly SizedRun[], cpu: string | null | undefined): number | null {
173187 const model = hostModel(cpu);
174188 const same = model ? onLarge.filter((r) => hostModel(r.cpu) === model) : [];
175- const secs = (same.length ? same : onLarge).map((r) => r.usage!.seconds).sort((a, b) => a - b);
176- return secs.length ? secs[Math.floor((secs.length - 1) / 2)]! : null;
189+ const secs = (same.length ? same : onLarge).slice(0, 10).map((r) => r.usage!.seconds).sort((a, b) => a - b);
190+ if (!secs.length) return null;
191+ const median = secs[Math.floor((secs.length - 1) / 2)]!;
192+ const spread = secs.length >= 5 ? secs[secs.length - 2]! : 0;
193+ return Math.max(median * SLOWER.factor + SLOWER.plusSeconds, spread);
177194 }
178195
179196 /** The log line naming the job's runner size and why it got it. */
@@ -206,5 +223,8 @@
206223 const busy = u.busySeconds;
207224 if (!Array.isArray(busy) || busy.length !== BUSY_BUCKETS.length || !busy.every((v) => typeof v === "number" && Number.isFinite(v) && v >= 0)) return null;
208225 out.busySeconds = busy.map((v: number) => Math.min(v, 1e6));
226+ // Optional: absent from runners before 2026-10-16 (such a run counts as not measured closely enough).
227+ const w = u.windowSeconds;
228+ if (typeof w === "number" && Number.isFinite(w) && w > 0) out.windowSeconds = Math.min(w, 60);
209229 return JSON.stringify(out);
210230 }
modifiedapps/diffinery/test/unit/runner-sizes.test.ts+37−4View file
@@ -4,7 +4,7 @@
44 import { parseWorkflow } from "@diffinery/workflow-yaml";
55 import { Actions } from "../../src/repo/actions.js";
66 import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
7-import { chooseSize, cleanUsage, containerName, describeSize, hostFactor, hostModel, sizeLabel, sizeOfContainer, peakCores, type SizedRun } from "../../src/runner/sizes.js";
7+import { chooseSize, cleanUsage, containerName, describeSize, hostFactor, hostModel, sizeLabel, sizeOfContainer, peakCores, slowLimit, type SizedRun } from "../../src/runner/sizes.js";
88 import { testD1 } from "./stubs/d1.js";
99 import type { RepoDO } from "../../src/repo/RepoDO.js";
1010 import type { Env } from "../../src/env.js";
@@ -24,6 +24,7 @@
2424 coresMax: 0,
2525 saturated: o.saturated ?? 0,
2626 busySeconds,
27+ windowSeconds: 0.25,
2728 memoryPeak: (o.memGiB ?? 0.3) * GiB,
2829 diskPeak: (o.diskGB ?? 1.5) * 1e9,
2930 cores: o.cores ?? 4,
@@ -114,9 +115,9 @@
114115 it("moves a job back to large after it took longer on a smaller runner", () => {
115116 const large = (seconds: number) => ran(usage({ seconds }));
116117 const history = [large(100), large(110), large(90), large(300)];
117- // Typical on large: 100 s (the median). 125 s on small is within 1.2x + 10 s; 131 s isn't.
118- expect(chooseSize([ran(usage({ seconds: 125 }), "tiny"), ...history]).size).toBe("tiny");
119- const c = chooseSize([ran(usage({ seconds: 131 }), "tiny"), ...history]);
118+ // Typical on large: 100 s (the median). 111 s on tiny is within a tenth and 2 s; 113 s isn't.
119+ expect(chooseSize([ran(usage({ seconds: 111 }), "tiny"), ...history]).size).toBe("tiny");
120+ const c = chooseSize([ran(usage({ seconds: 113 }), "tiny"), ...history]);
120121 expect(c.size).toBe("large");
121122 expect(c.why).toContain("took longer on a tiny runner");
122123 // Large runs further back still give the typical time.
@@ -141,6 +142,38 @@
141142 expect(chooseSize([ran(usage(), "large", "success", null), ran(usage()), ran(usage())]).size).toBe("small");
142143 });
143144
145+ it("allows for a job's own noise on large, and no more", () => {
146+ const large = (seconds: number[]) => seconds.map((s) => ran(usage({ seconds: s })));
147+ // A lint job that took 9 to 14 s on large (median 11): 14.1 s is the limit.
148+ expect(slowLimit(large([9, 10, 11, 12, 14]), ZEN2)).toBeCloseTo(14.1, 5);
149+ // 15 s on tiny is slower than any of them: back to large. 13 s on medium is within its noise.
150+ const lint = large([9, 10, 11, 12, 14]);
151+ expect(chooseSize([ran(usage({ seconds: 15 }), "tiny"), ...lint]).size).toBe("large");
152+ expect(chooseSize([ran(usage({ seconds: 13 }), "medium"), ...lint]).size).not.toBe("large");
153+ // A job whose large runs vary a lot (a deploy waiting on another service) is judged by its slowest but one.
154+ expect(slowLimit(large([45, 60, 80, 120, 150]), ZEN2)).toBe(120);
155+ // Fewer than 5 runs: the median rule alone.
156+ expect(slowLimit(large([45, 150]), ZEN2)).toBeCloseTo(51.5, 5);
157+ expect(slowLimit([], ZEN2)).toBeNull();
158+ });
159+
160+ it("counts runs measured over 2-second windows as not measured", () => {
161+ const coarse = { ...usage(), windowSeconds: 2 };
162+ const old = { ...usage() } as JobUsage;
163+ delete old.windowSeconds;
164+ expect(chooseSize(runs(ran(coarse))).why).toContain("closely enough");
165+ expect(chooseSize(runs(ran(old))).size).toBe("large");
166+ });
167+
168+ it("keeps a job with short single-core bursts on a runner with a whole core to spare", () => {
169+ // Ten bursts of a third of a second on one core in 30 s, as quarter-second windows see them: about
170+ // 2.5 s at 1 core. Over 2-second windows the same job read as a sixth of a core and got half a vCPU.
171+ const bursty = usage({ seconds: 30, busy: { 0.1: 27.5, 1: 2.5 } });
172+ expect(chooseSize(runs(ran(bursty), ran(bursty), ran(bursty))).size).toBe("medium");
173+ const onZen4 = ran(bursty, "large", "success", ZEN4);
174+ expect(chooseSize([onZen4, onZen4, onZen4]).size).toBe("large");
175+ });
176+
144177 it("compares a run's time with large runs on the same host processor", () => {
145178 const at = (seconds: number, size: SizedRun["size"], cpu: string) => ran(usage({ seconds }), size, "success", cpu);
146179 // Large: 60 s on Zen 4, 120 s on Zen 2. 110 s on tiny on Zen 4 is slower than Zen 4's 60 s, though
modifiedcontainers/runner/src/usage.ts+5−2View file
@@ -50,8 +50,10 @@
5050 constructor(o: UsageSamplerOptions = {}) {
5151 this.proc = o.procRoot ?? "/proc";
5252 this.diskPath = o.diskPath ?? "/";
53- this.sampleMs = o.sampleMs ?? 500;
54- this.windowMs = o.windowMs ?? 2000;
53+ // A quarter second: a short burst of CPU is measured at its height, not averaged away
54+ // (apps/diffinery/src/runner/sizes.ts sizes later runs by the busiest window).
55+ this.sampleMs = o.sampleMs ?? 250;
56+ this.windowMs = o.windowMs ?? 250;
5557 this.visible = o.visible ?? os.availableParallelism();
5658 this.cores = o.cores && o.cores > 0 ? o.cores : this.visible;
5759 }
@@ -86,6 +88,7 @@
8688 coresMax: round(sorted.length ? sorted[sorted.length - 1]! : 0),
8789 saturated: round(sorted.length ? sorted.filter((b) => b >= saturatedAt(this.cores)).length / sorted.length : 0, 3),
8890 busySeconds: this.busySeconds.map((s) => round(s, 1)),
91+ windowSeconds: this.windowMs / 1000,
8992 memoryPeak: this.memPeak,
9093 diskPeak: this.diskPeak,
9194 cores: this.cores,
modifiedcontainers/runner/test/usage.test.ts+28−0View file
@@ -17,6 +17,34 @@
1717 }
1818
1919 describe("usage sampler", () => {
20+ it("measures a short burst at its height with its default quarter-second windows", () => {
21+ vi.useFakeTimers({ now: 0 });
22+ try {
23+ for (const [windowMs, expected] of [[undefined, 1], [2000, 0.13]] as const) {
24+ const proc = path.join(tmp(), "proc");
25+ const set = fakeProc(proc);
26+ let busy = 10;
27+ let idle = 0;
28+ set(busy, idle, 500);
29+ const s = new UsageSampler({ procRoot: proc, diskPath: proc, visible: 4, ...(windowMs ? { windowMs, sampleMs: 500 } : {}) });
30+ s.start();
31+ // 1.75 s idle, then a quarter second with one of four CPUs busy, then idle: 10 ms ticks.
32+ for (let t = 0; t < 400; t++) {
33+ const on = t >= 175 && t < 200;
34+ busy += on ? 1 : 0;
35+ idle += on ? 3 : 4;
36+ set(busy, idle, 500);
37+ vi.advanceTimersByTime(10);
38+ }
39+ const u = s.stop()!;
40+ expect(u.coresMax).toBeCloseTo(expected, 1);
41+ expect(u.windowSeconds).toBe((windowMs ?? 250) / 1000);
42+ }
43+ } finally {
44+ vi.useRealTimers();
45+ }
46+ });
47+
2048 it("measures busy cores per window, peak memory and OOM kills", () => {
2149 vi.useFakeTimers({ now: 0 });
2250 try {
modifiedpackages/shared/src/runner.ts+3−1View file
@@ -146,13 +146,15 @@
146146 /** Seconds measured, and CPU seconds used (all cores) in that time. */
147147 seconds: number;
148148 cpuSeconds: number;
149- /** Busy cores at the 95th percentile and at the most busy sample (2 s windows). */
149+ /** Busy cores at the 95th percentile and in the busiest window (see windowSeconds). */
150150 coresP95: number;
151151 coresMax: number;
152152 /** Share of samples (0 to 1) in which the job kept nearly all its cores busy (90%, or all but a quarter core): it was held back by the CPU. */
153153 saturated: number;
154154 /** Seconds spent at each level of busy cores: entry i counts windows with at most BUSY_BUCKETS[i] cores busy (and more than the one before). */
155155 busySeconds: number[];
156+ /** How long the windows busy cores were measured over, in seconds (absent from runners before 2026-10-16, which used 2 s). */
157+ windowSeconds?: number;
156158 /** Peak memory and disk in use, in bytes. */
157159 memoryPeak: number;
158160 diskPeak: number;