Runner sizes: account for the host processor, and don't move back and forth (#204)
* Runner sizes: account for the host processor, and don't move back and forth
- CI / Typecheck and bundlesuccess
- CI / Unit tests (1/2)success
- CI / Unit tests (2/2)success
- Deploy / Deploy to Cloudflaresuccess
- Deploy / Main's current tipsuccess
- End-to-end / End-to-end (agent reviews, tasks, Today)success
- End-to-end / End-to-end (git, PRs, Actions)success
- End-to-end / End-to-end (large pushes)success
- End-to-end / End-to-end (passkeys, registry, email)success
- End-to-end / End-to-end (real histories)success
- End-to-end / End-to-end (security)success
- Live test / Containers, registry, sandboxes, webhooks, schedules, LFS and pull requestssuccess
4 files changed+160−57
Files (4)
modifiedapps/diffinery/docs/actions/runners.md+2−2View file
| @@ -89,8 +89,8 @@ | ||
| 89 | 89 | How a size is picked for a job without a label: |
| 90 | 90 | |
| 91 | 91 | - The runner measures every hosted job: busy cores every 2 seconds, memory and disk in use every half second, and processes the kernel killed for want of memory. The last lines of the job's log say what it used, for example `Resources: 0.1 of 4 vCPUs busy on average (0.3 at the 95th percentile), peak memory 0.3 GiB of 12.0 GiB, peak disk 1.6 GB of 19.3 GB`. The job API (`GET /repos/{owner}/{repo}/actions/jobs/{job_id}`) returns it as `usage`, and the size as `runner_size`. |
| 92 | -- The next run of the same job (the same workflow file and job name, so each matrix leg on its own) looks at up to its last 5 runs, and needs at least 3. It gets a smaller size only if every one of them would have fit it with room to spare: peak memory times 1.5, plus 0.5 GiB, and peak disk times 1.25, plus 1 GB, within the size; and at least half as many vCPUs again as the job kept busy in its busiest 2-second window, so a job that bursts onto several cores even briefly, such as a test suite, keeps a runner with room for that. A job that kept its runner's CPU busy to the limit never gets a smaller size than that one. | |
| 93 | -- A job stays on large when it has no measured runs yet, when one of those runs ran out of memory, when one failed or timed out on a smaller size, or when one took longer on a smaller size than the job typically takes on large (its median, plus a fifth and 10 seconds). A re-run always runs on large, and so does a job that found no smaller runner free when it first started. So a smaller size is never the reason a job fails or slows down twice. | |
| 92 | +- The next run of the same job (the same workflow file and job name, so each matrix leg on its own) looks at up to its last 5 runs, and needs at least 3. It gets a smaller size only if every one of them would have fit it with room to spare: peak memory times 1.5, plus 0.5 GiB, and peak disk times 1.25, plus 1 GB, within the size; and at least half as many vCPUs again as the job kept busy in its busiest 2-second window, so a job that bursts onto several cores even briefly, such as a test suite, keeps a runner with room for that. Busy cores count as on the slowest host processor runners land on: Cloudflare's hosts mix AMD EPYC generations whose cores differ about 2x in speed, so a job measured on a fast host gets cores to spare for a slow one (a run whose host processor isn't known counts as on the fastest). A job that kept its runner's CPU busy to the limit never gets a smaller size than that one. | |
| 93 | +- A job stays on large when it has no measured runs yet, when one of those runs ran out of memory, when one failed or timed out on a smaller size, or when one took longer on a smaller size than the job typically takes on large (its median on the same host processor where there is one, plus a fifth and 10 seconds). After a smaller size ran a job out of memory, failed it or slowed it, the job stays on large for its next 20 runs, so it doesn't move back and forth. A re-run always runs on large, and so does a job that found no smaller runner free when it first started. So a smaller size is never the reason a job fails or slows down twice. | |
| 94 | 94 | - The job's log names the size it got and why, near the top: `Runner size: tiny (0.25 of a vCPU, 1 GiB of memory, 4 GB of disk), sized by this job's last 5 runs (peak memory 0.3 GiB)`. |
| 95 | 95 | |
| 96 | 96 | Set `runs-on: diffinery-large` to keep a job on large whatever it uses. |
modifiedapps/diffinery/src/repo/actions.ts+35−21View file
| @@ -37,7 +37,7 @@ | ||
| 37 | 37 | import { schedulerStub } from "../lib/schedules.js"; |
| 38 | 38 | import { APPROVAL_DAYS, type Deployment, type DeploymentReview, type Reviewer } from "./deployments.js"; |
| 39 | 39 | import { referencedSecrets, RUNNER_SECRETS, secretsFor } from "./job-secrets.js"; |
| 40 | -import { chooseSize, cleanUsage, containerName, describeSize, RUNNER_SIZES, sizeLabel, sizeOfContainer, type RunnerSize, type SizedRun } from "../runner/sizes.js"; | |
| 40 | +import { chooseSize, cleanUsage, containerName, describeSize, LOOKBACK, RUNNER_SIZES, sizeLabel, sizeOfContainer, type RunnerSize, type SizedRun } from "../runner/sizes.js"; | |
| 41 | 41 | |
| 42 | 42 | /** How long a sandbox token works after the sandbox starts. */ |
| 43 | 43 | export const SANDBOX_TOKEN_TTL_MS = 24 * 3600_000; |
| @@ -1274,26 +1274,40 @@ | ||
| 1274 | 1274 | size = pinned; |
| 1275 | 1275 | why = "asked for by runs-on"; |
| 1276 | 1276 | } else { |
| 1277 | - const rows = this.sql | |
| 1278 | - .exec<{ conclusion: string | null; container: string; usage: string | null }>( | |
| 1279 | - `SELECT j.conclusion, j.container, j.usage FROM jobs j JOIN runs r ON r.id = j.run_id | |
| 1280 | - WHERE r.workflow_path = ? AND j.name = ? AND j.id != ? AND j.status = 'completed' AND j.labels IS NULL AND j.container IS NOT NULL AND j.carried_from IS NULL | |
| 1281 | - ORDER BY j.completed_at DESC LIMIT 20`, | |
| 1282 | - run.workflow_path, | |
| 1283 | - job.name, | |
| 1284 | - job.id, | |
| 1285 | - ) | |
| 1286 | - .toArray(); | |
| 1287 | - const history: SizedRun[] = rows.map((r) => { | |
| 1288 | - let usage: SizedRun["usage"] = null; | |
| 1289 | - try { | |
| 1290 | - usage = r.usage ? JSON.parse(r.usage) : null; | |
| 1291 | - } catch { | |
| 1292 | - /* unreadable: counts as not measured */ | |
| 1293 | - } | |
| 1294 | - return { conclusion: r.conclusion, size: sizeOfContainer(r.container), usage }; | |
| 1295 | - }); | |
| 1296 | - ({ size, why } = chooseSize(history, { rerun: run.attempt > 1 })); | |
| 1277 | + type Row = { conclusion: string | null; container: string; usage: string | null; runner_info: string | null }; | |
| 1278 | + const hosted = `FROM jobs j JOIN runs r ON r.id = j.run_id | |
| 1279 | + WHERE r.workflow_path = ? AND j.name = ? AND j.id != ? AND j.status = 'completed' AND j.labels IS NULL AND j.container IS NOT NULL AND j.carried_from IS NULL`; | |
| 1280 | + const sized = (rows: Row[]): SizedRun[] => | |
| 1281 | + rows.map((r) => { | |
| 1282 | + let usage: SizedRun["usage"] = null; | |
| 1283 | + let cpu: string | null = null; | |
| 1284 | + try { | |
| 1285 | + usage = r.usage ? JSON.parse(r.usage) : null; | |
| 1286 | + } catch { | |
| 1287 | + /* unreadable: counts as not measured */ | |
| 1288 | + } | |
| 1289 | + try { | |
| 1290 | + cpu = r.runner_info ? ((JSON.parse(r.runner_info) as { cpu?: unknown }).cpu as string | undefined) ?? null : null; | |
| 1291 | + } catch { | |
| 1292 | + /* unreadable: an unknown host */ | |
| 1293 | + } | |
| 1294 | + return { conclusion: r.conclusion, size: sizeOfContainer(r.container), usage, cpu: typeof cpu === "string" ? cpu : null }; | |
| 1295 | + }); | |
| 1296 | + const history = sized( | |
| 1297 | + this.sql.exec<Row>(`SELECT j.conclusion, j.container, j.usage, j.runner_info ${hosted} ORDER BY j.completed_at DESC LIMIT ${LOOKBACK}`, run.workflow_path, job.name, job.id).toArray(), | |
| 1298 | + ); | |
| 1299 | + // Its typical time on large, from its last measured large runs however far back they are. | |
| 1300 | + const large = sized( | |
| 1301 | + this.sql | |
| 1302 | + .exec<Row>( | |
| 1303 | + `SELECT j.conclusion, j.container, j.usage, j.runner_info ${hosted} AND j.conclusion = 'success' AND j.usage IS NOT NULL AND j.container NOT LIKE '%.%' ORDER BY j.completed_at DESC LIMIT 10`, | |
| 1304 | + run.workflow_path, | |
| 1305 | + job.name, | |
| 1306 | + job.id, | |
| 1307 | + ) | |
| 1308 | + .toArray(), | |
| 1309 | + ); | |
| 1310 | + ({ size, why } = chooseSize(history, { rerun: run.attempt > 1, large })); | |
| 1297 | 1311 | // A job that already waited for a machine doesn't wait again for a smaller one. |
| 1298 | 1312 | if (size !== "large" && job.placement_tries > 0) { |
| 1299 | 1313 | size = "large"; |
modifiedapps/diffinery/src/runner/sizes.ts+68−24View file
| @@ -45,9 +45,36 @@ | ||
| 45 | 45 | return (m?.[1] as RunnerSize | undefined) ?? "large"; |
| 46 | 46 | } |
| 47 | 47 | |
| 48 | -/** One earlier run of the same job: how it ended, on which size, and what it used. */ | |
| 49 | -export type SizedRun = { conclusion: string | null; size: RunnerSize; usage: JobUsage | null }; | |
| 48 | +/** | |
| 49 | + * One earlier run of the same job: how it ended, on which size, what it used, and the host processor it | |
| 50 | + * ran on as the runner reported it (`Host:` in the log, e.g. "AMD EPYC 9654 96-Core Processor (family 25, model 17), ..."). | |
| 51 | + */ | |
| 52 | +export type SizedRun = { conclusion: string | null; size: RunnerSize; usage: JobUsage | null; cpu?: string | null }; | |
| 50 | 53 | |
| 54 | +/** | |
| 55 | + * How long a fixed single-thread loop took on each host processor Cloudflare's runners landed on (cpu | |
| 56 | + * family/model; scalability probe, 2026-10-16): Zen 4 623 ms, Zen 3 837 ms at best, Zen 2 1190 ms; the slowest | |
| 57 | + * host seen took SLOWEST_HOST_MS. A core on a slow host does about half the work of one on a fast host. | |
| 58 | + */ | |
| 59 | +const HOST_LOOP_MS: Record<string, number> = { "23/49": 1190, "25/1": 837, "25/17": 623 }; | |
| 60 | +export const SLOWEST_HOST_MS = 1362; | |
| 61 | + | |
| 62 | +/** A host processor's family and model ("25/17"), from the runner's description of it; null when it doesn't say. */ | |
| 63 | +export function hostModel(cpu: string | null | undefined): string | null { | |
| 64 | + const m = /\(family (\d+), model (\d+)\)/.exec(cpu ?? ""); | |
| 65 | + return m ? `${m[1]}/${m[2]}` : null; | |
| 66 | +} | |
| 67 | + | |
| 68 | +/** | |
| 69 | + * How many times the cores a run kept busy it would keep busy for the same work on the slowest host. Where | |
| 70 | + * a job lands isn't ours to pick, so a smaller runner must serve it there too; an unknown processor | |
| 71 | + * counts as the fastest one, which gives the largest factor. | |
| 72 | + */ | |
| 73 | +export function hostFactor(cpu: string | null | undefined): number { | |
| 74 | + const ms = HOST_LOOP_MS[hostModel(cpu) ?? ""] ?? Math.min(...Object.values(HOST_LOOP_MS)); | |
| 75 | + return Math.max(1, SLOWEST_HOST_MS / ms); | |
| 76 | +} | |
| 77 | + | |
| 51 | 78 | /** Headroom a smaller runner must leave over what the job used: memory and disk are sampled, so a peak can be missed. */ |
| 52 | 79 | export const MEMORY_HEADROOM = { factor: 1.5, plusGiB: 0.5 }; |
| 53 | 80 | export const DISK_HEADROOM = { factor: 1.25, plusGB: 1 }; |
| @@ -58,7 +85,8 @@ | ||
| 58 | 85 | export const CPU_HEADROOM = 1.5; |
| 59 | 86 | /** |
| 60 | 87 | * A run on a smaller runner that took longer than this over the job's typical time on large (the median of |
| 61 | - * its measured large runs) was slowed by it, or may have been: the job goes back to large. | |
| 88 | + * its measured large runs, on the same host processor when there are any) was slowed by it, or may have | |
| 89 | + * been: the job goes back to large. | |
| 62 | 90 | */ |
| 63 | 91 | export const SLOWER = { factor: 1.2, plusSeconds: 10 }; |
| 64 | 92 | /** A run busy at its runner's limit this share of its time may have wanted more: no smaller runner than that one. */ |
| @@ -66,6 +94,11 @@ | ||
| 66 | 94 | /** Earlier runs looked at, and how many measured ones a job needs before it can get a smaller runner. */ |
| 67 | 95 | export const HISTORY = 5; |
| 68 | 96 | export const MIN_MEASURED = 3; |
| 97 | +/** | |
| 98 | + * Once a smaller runner ran a job out of memory, failed it or slowed it, the job stays on large while that | |
| 99 | + * run is among its last LOOKBACK, so it doesn't move back and forth between sizes. | |
| 100 | + */ | |
| 101 | +export const LOOKBACK = 20; | |
| 69 | 102 | |
| 70 | 103 | /** |
| 71 | 104 | * The most cores a run kept busy in any 2-second window, counted at its level's upper bound (never below |
| @@ -88,7 +121,8 @@ | ||
| 88 | 121 | if (u.diskPeak / 1e9 * DISK_HEADROOM.factor + DISK_HEADROOM.plusGB > s.diskGB) return "disk"; |
| 89 | 122 | // Held back by its own runner's CPU: it may need more than it showed, so nothing smaller than that runner. |
| 90 | 123 | if (u.saturated > SATURATED && s.vcpu <= RUNNER_SIZES[r.size].vcpu) return "cpu"; |
| 91 | - if (peakCores(u) * CPU_HEADROOM > s.vcpu) return "cpu"; | |
| 124 | + // Counted as on the slowest host, with headroom for bursts shorter than a sample. | |
| 125 | + if (peakCores(u) * hostFactor(r.cpu) * CPU_HEADROOM > s.vcpu) return "cpu"; | |
| 92 | 126 | return null; |
| 93 | 127 | } |
| 94 | 128 | |
| @@ -96,30 +130,29 @@ | ||
| 96 | 130 | |
| 97 | 131 | /** |
| 98 | 132 | * The size for a hosted job from its earlier runs (the same workflow and job name, newest first). It is |
| 99 | - * `large` unless each of its last few runs (at least MIN_MEASURED, all measured) fits a smaller one with headroom: peak memory | |
| 100 | - * and disk well inside it, and half as many cores again as it kept busy at its busiest. A run that ran out | |
| 101 | - * of memory, failed on a smaller runner, took longer there than the job takes on large, or wasn't measured | |
| 102 | - * keeps the job on `large`, so a smaller runner is never what makes a job fail or slow down twice. A re-run | |
| 103 | - * of a failed job always gets `large`. Runs past the last few only give the job's typical time on large. | |
| 133 | + * `large` unless each of its last few runs (at least MIN_MEASURED, all measured) fits a smaller one with | |
| 134 | + * headroom: peak memory and disk well inside it, and half as many cores again as it kept busy at its | |
| 135 | + * busiest, counted as on the slowest host. A run that ran out of memory, failed on a smaller runner, or | |
| 136 | + * took longer there than the job takes on large keeps the job on `large` for its next LOOKBACK runs, so a | |
| 137 | + * smaller runner is never what makes a job fail or slow down twice. A re-run of a failed job always gets | |
| 138 | + * `large`. `large` gives the job's measured runs on large when they go back further than `history`. | |
| 104 | 139 | */ |
| 105 | -export function chooseSize(history: readonly SizedRun[], o: { rerun?: boolean } = {}): SizeChoice { | |
| 140 | +export function chooseSize(history: readonly SizedRun[], o: { rerun?: boolean; large?: readonly SizedRun[] } = {}): SizeChoice { | |
| 106 | 141 | if (o.rerun) return { size: "large", why: "a re-run runs on the large runner" }; |
| 107 | - const runs = history.filter((r) => r.conclusion !== "cancelled" && r.conclusion !== "skipped").slice(0, HISTORY); | |
| 142 | + const counted = history.filter((r) => r.conclusion !== "cancelled" && r.conclusion !== "skipped").slice(0, LOOKBACK); | |
| 143 | + const runs = counted.slice(0, HISTORY); | |
| 108 | 144 | if (!runs.length) return { size: "large", why: "no earlier run of this job to size it by" }; |
| 109 | - for (const r of runs) { | |
| 110 | - if (!r.usage) return { size: "large", why: "an earlier run of this job wasn't measured" }; | |
| 111 | - if (r.usage.oomKills) return { size: "large", why: "an earlier run of this job ran out of memory" }; | |
| 112 | - if (r.size !== "large" && r.conclusion !== "success") return { size: "large", why: `an earlier run of this job failed on a ${r.size} runner` }; | |
| 145 | + for (const r of runs) if (!r.usage) return { size: "large", why: "an earlier run of this job wasn't measured" }; | |
| 146 | + const onLarge = (o.large ?? history).filter((r) => r.size === "large" && r.conclusion === "success" && r.usage); | |
| 147 | + for (const r of counted) { | |
| 148 | + if (r.usage?.oomKills) return { size: "large", why: "an earlier run of this job ran out of memory" }; | |
| 149 | + if (r.size === "large") continue; | |
| 150 | + if (r.conclusion !== "success") return { size: "large", why: `an earlier run of this job failed on a ${r.size} runner` }; | |
| 151 | + const typical = typicalSeconds(onLarge, r.cpu); | |
| 152 | + if (r.usage && typical !== null && r.usage.seconds > typical * SLOWER.factor + SLOWER.plusSeconds) { | |
| 153 | + return { size: "large", why: `an earlier run of this job took longer on a ${r.size} runner than it does on large` }; | |
| 154 | + } | |
| 113 | 155 | } |
| 114 | - const onLarge = history | |
| 115 | - .filter((r) => r.size === "large" && r.conclusion === "success" && r.usage) | |
| 116 | - .map((r) => r.usage!.seconds) | |
| 117 | - .sort((a, b) => a - b); | |
| 118 | - if (onLarge.length) { | |
| 119 | - const typical = onLarge[Math.floor((onLarge.length - 1) / 2)]!; | |
| 120 | - const slow = runs.find((r) => r.size !== "large" && r.usage!.seconds > typical * SLOWER.factor + SLOWER.plusSeconds); | |
| 121 | - if (slow) return { size: "large", why: `an earlier run of this job took longer on a ${slow.size} runner than it does on large` }; | |
| 122 | - } | |
| 123 | 156 | if (runs.length < MIN_MEASURED) return { size: "large", why: `fewer than ${MIN_MEASURED} measured runs of this job to size it by` }; |
| 124 | 157 | const measured = runs as Array<SizedRun & { usage: JobUsage }>; |
| 125 | 158 | for (const size of ASCENDING) { |
| @@ -132,6 +165,17 @@ | ||
| 132 | 165 | return { size: "large", why: "this job uses what a smaller runner doesn't have" }; |
| 133 | 166 | } |
| 134 | 167 | |
| 168 | +/** | |
| 169 | + * A job's typical time on large: the median of its measured large runs on the same host processor as `cpu` | |
| 170 | + * (cores differ about 2x between hosts), or of all of them when none ran on that one; null with none. | |
| 171 | + */ | |
| 172 | +function typicalSeconds(onLarge: readonly SizedRun[], cpu: string | null | undefined): number | null { | |
| 173 | + const model = hostModel(cpu); | |
| 174 | + const same = model ? onLarge.filter((r) => hostModel(r.cpu) === model) : []; | |
| 175 | + const secs = (same.length ? same : onLarge).map((r) => r.usage!.seconds).sort((a, b) => a - b); | |
| 176 | + return secs.length ? secs[Math.floor((secs.length - 1) / 2)]! : null; | |
| 177 | +} | |
| 178 | + | |
| 135 | 179 | /** The log line naming the job's runner size and why it got it. */ |
| 136 | 180 | export function describeSize(size: RunnerSize, why: string, pinned: boolean): string { |
| 137 | 181 | const s = RUNNER_SIZES[size]; |
modifiedapps/diffinery/test/unit/runner-sizes.test.ts+55−10View file
| @@ -4,7 +4,7 @@ | ||
| 4 | 4 | import { parseWorkflow } from "@diffinery/workflow-yaml"; |
| 5 | 5 | import { Actions } from "../../src/repo/actions.js"; |
| 6 | 6 | import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js"; |
| 7 | -import { chooseSize, cleanUsage, containerName, describeSize, sizeLabel, sizeOfContainer, peakCores, type SizedRun } from "../../src/runner/sizes.js"; | |
| 7 | +import { chooseSize, cleanUsage, containerName, describeSize, hostFactor, hostModel, sizeLabel, sizeOfContainer, peakCores, type SizedRun } from "../../src/runner/sizes.js"; | |
| 8 | 8 | import { testD1 } from "./stubs/d1.js"; |
| 9 | 9 | import type { RepoDO } from "../../src/repo/RepoDO.js"; |
| 10 | 10 | import type { Env } from "../../src/env.js"; |
| @@ -32,7 +32,11 @@ | ||
| 32 | 32 | oomKills: o.oom === undefined ? 0 : o.oom, |
| 33 | 33 | }; |
| 34 | 34 | } |
| 35 | -const ran = (u: JobUsage | null, size: SizedRun["size"] = "large", conclusion = "success"): SizedRun => ({ conclusion, size, usage: u }); | |
| 35 | +/** Host processors as the runner describes them (`Host:` lines seen on 2026-10-16). */ | |
| 36 | +const ZEN2 = "AMD EPYC 7B12 64-Core Processor (family 23, model 49), 4 vCPUs, 2250 MHz"; | |
| 37 | +const ZEN4 = "AMD EPYC 9654 96-Core Processor (family 25, model 17), 4 vCPUs, 2400 MHz"; | |
| 38 | +/** A run, on a Zen 2 host (the slowest generation, so busy cores count nearly as measured) unless `cpu` says otherwise. */ | |
| 39 | +const ran = (u: JobUsage | null, size: SizedRun["size"] = "large", conclusion = "success", cpu: string | null = ZEN2): SizedRun => ({ conclusion, size, usage: u, cpu }); | |
| 36 | 40 | /** The run, then enough measured runs like it behind it to size by. */ |
| 37 | 41 | const runs = (...first: SizedRun[]) => [...first, ...Array.from({ length: Math.max(0, 3 - first.length) }, () => ran(usage()))]; |
| 38 | 42 | |
| @@ -93,7 +97,7 @@ | ||
| 93 | 97 | // The peak reported by the runner counts even where the buckets show less, with half as much again |
| 94 | 98 | // to spare: 0.4 cores needs 0.6, more than small's half vCPU. |
| 95 | 99 | expect(chooseSize(runs(ran({ ...usage(), coresMax: 0.4 }))).size).toBe("medium"); |
| 96 | - expect(chooseSize(runs(ran({ ...usage(), coresMax: 0.3 }))).size).toBe("small"); | |
| 100 | + expect(chooseSize(runs(ran({ ...usage(), coresMax: 0.25 }))).size).toBe("small"); | |
| 97 | 101 | // Steady single-core work fits medium's 2 vCPU but not half a vCPU. |
| 98 | 102 | expect(chooseSize(runs(ran(usage({ seconds: 200, busy: { 1: 200 }, memGiB: 1 })))).size).toBe("medium"); |
| 99 | 103 | }); |
| @@ -120,6 +124,46 @@ | ||
| 120 | 124 | expect(chooseSize(old).size).toBe("large"); |
| 121 | 125 | }); |
| 122 | 126 | |
| 127 | + it("counts busy cores as on the slowest host, since a job can land there", () => { | |
| 128 | + expect(hostModel(ZEN4)).toBe("25/17"); | |
| 129 | + expect(hostModel("Intel something")).toBeNull(); | |
| 130 | + expect(hostFactor(ZEN2)).toBeCloseTo(1.14, 2); | |
| 131 | + expect(hostFactor(ZEN4)).toBeCloseTo(2.19, 2); | |
| 132 | + // An unknown processor counts as the fastest. | |
| 133 | + expect(hostFactor(null)).toBe(hostFactor(ZEN4)); | |
| 134 | + // 0.25 cores on Zen 2 fits small's half vCPU with headroom; the same 0.25 cores on Zen 4 is about 0.55 | |
| 135 | + // on the slowest host, so medium. | |
| 136 | + const quarter = usage({ busy: { 0.25: 120 } }); | |
| 137 | + expect(chooseSize(runs(ran(quarter), ran(quarter), ran(quarter))).size).toBe("small"); | |
| 138 | + const fast = ran(quarter, "large", "success", ZEN4); | |
| 139 | + expect(chooseSize([fast, ran(quarter), ran(quarter)]).size).toBe("medium"); | |
| 140 | + // Unknown hosts: a mostly idle job gets small, not tiny. | |
| 141 | + expect(chooseSize([ran(usage(), "large", "success", null), ran(usage()), ran(usage())]).size).toBe("small"); | |
| 142 | + }); | |
| 143 | + | |
| 144 | + it("compares a run's time with large runs on the same host processor", () => { | |
| 145 | + const at = (seconds: number, size: SizedRun["size"], cpu: string) => ran(usage({ seconds }), size, "success", cpu); | |
| 146 | + // Large: 60 s on Zen 4, 120 s on Zen 2. 110 s on tiny on Zen 4 is slower than Zen 4's 60 s, though | |
| 147 | + // faster than the median of all large runs. | |
| 148 | + const large = [at(60, "large", ZEN4), at(60, "large", ZEN4), at(120, "large", ZEN2), at(120, "large", ZEN2), at(120, "large", ZEN2)]; | |
| 149 | + expect(chooseSize([at(110, "tiny", ZEN4), ...large]).size).toBe("large"); | |
| 150 | + // 130 s on tiny on Zen 2 is like Zen 2's large runs: no slowdown (small, since some runs were on Zen 4). | |
| 151 | + expect(chooseSize([at(130, "tiny", ZEN2), ...large]).size).toBe("small"); | |
| 152 | + // Large runs from further back (o.large) give the typical time when the last runs are all smaller. | |
| 153 | + const smallOnly = Array.from({ length: 5 }, () => at(110, "tiny", ZEN4)); | |
| 154 | + expect(chooseSize(smallOnly).size).toBe("small"); | |
| 155 | + expect(chooseSize(smallOnly, { large }).size).toBe("large"); | |
| 156 | + }); | |
| 157 | + | |
| 158 | + it("stays large for 20 runs after a smaller runner hurt a job, so it doesn't flap", () => { | |
| 159 | + const fine = () => ran(usage()); | |
| 160 | + const failed = ran(usage(), "tiny", "failure"); | |
| 161 | + // The failure is 10 runs back: still large. | |
| 162 | + expect(chooseSize([...Array.from({ length: 10 }, fine), failed]).size).toBe("large"); | |
| 163 | + // 20 runs later it has dropped out of view. | |
| 164 | + expect(chooseSize([...Array.from({ length: 20 }, fine), failed]).size).toBe("tiny"); | |
| 165 | + }); | |
| 166 | + | |
| 123 | 167 | it("doesn't go below a runner whose CPU held the job back", () => { |
| 124 | 168 | // Busy at small's half vCPU most of the time: it may need more, so at least medium. |
| 125 | 169 | const capped = usage({ seconds: 100, busy: { 0.5: 100 }, cores: 0.5, saturated: 0.9 }); |
| @@ -236,15 +280,16 @@ | ||
| 236 | 280 | } |
| 237 | 281 | |
| 238 | 282 | const second = await f.run(); |
| 239 | - expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER_TINY", name: `job-${second}.tiny`, vcpu: 0.25 }); | |
| 240 | - expect(f.actions.getJob(second).runnerSize).toBe("tiny"); | |
| 241 | - expect(f.job(second).size_note).toMatch(/^Runner size: tiny/); | |
| 283 | + // The runs reported no host processor, so they count as on the fastest: small, not tiny. | |
| 284 | + expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER_SMALL", name: `job-${second}.small`, vcpu: 0.5 }); | |
| 285 | + expect(f.actions.getJob(second).runnerSize).toBe("small"); | |
| 286 | + expect(f.job(second).size_note).toMatch(/^Runner size: small/); | |
| 242 | 287 | |
| 243 | 288 | // It fails there: the next run is large again. |
| 244 | - await f.complete(second, "failure", usage({ memGiB: 0.3, cores: 0.25 })); | |
| 289 | + await f.complete(second, "failure", usage({ memGiB: 0.3, cores: 0.5 })); | |
| 245 | 290 | const third = await f.run(); |
| 246 | 291 | expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER", name: `job-${third}` }); |
| 247 | - expect(f.job(third).size_note).toContain("failed on a tiny runner"); | |
| 292 | + expect(f.job(third).size_note).toContain("failed on a small runner"); | |
| 248 | 293 | }); |
| 249 | 294 | |
| 250 | 295 | it("runs a job where runs-on asks, and stops it there", async () => { |
| @@ -258,10 +303,10 @@ | ||
| 258 | 303 | }); |
| 259 | 304 | |
| 260 | 305 | it("starts a job large when no smaller runner was free for it", async () => { |
| 261 | - const f = fixture(undefined, (b) => b === "RUNNER_TINY"); | |
| 306 | + const f = fixture(undefined, (b) => b === "RUNNER_SMALL"); | |
| 262 | 307 | for (let i = 0; i < 3; i++) await f.complete(await f.run(), "success", usage({ memGiB: 0.3 })); |
| 263 | 308 | const second = await f.run(); |
| 264 | - // The tiny runner had no machine: the job went back in line, and starts again on large. | |
| 309 | + // The small runner had no machine: the job went back in line, and starts again on large. | |
| 265 | 310 | expect(f.job(second)).toMatchObject({ container: null }); |
| 266 | 311 | await f.actions.relaunchJob(second); |
| 267 | 312 | expect(f.started.at(-1)).toMatchObject({ binding: "RUNNER", name: `job-${second}` }); |