CI: test verdicts that don't depend on which host a runner lands on (#203)

* CI: test verdicts that don't depend on which host a runner lands on

test-bot authored 10 days ago1 parent c74f242commit e14fcb18a6790fc06cf8119ce8e7487ec6a0faa8Browse files

CI / Typecheck and bundlesuccess
CI / Unit tests (1/2)success
CI / Unit tests (2/2)success
Deploy / Deploy to Cloudflaresuccess
Deploy / Main's current tipsuccess
End-to-end / End-to-end (agent reviews, tasks, Today)Failed
End-to-end / End-to-end (git, PRs, Actions)success
End-to-end / End-to-end (large pushes)success
End-to-end / End-to-end (passkeys, registry, email)Failed
End-to-end / End-to-end (real histories)success
End-to-end / End-to-end (security)Failed

15 files changed+134−28

Files (15)

apps/cli/vitest.config.ts+4−0
apps/diffinery/docs/actions/runners.md+1−1
apps/diffinery/src/repo/actions.ts+3−3
apps/diffinery/test/unit/actions-runtime.test.ts+1−1
apps/diffinery/vitest.config.ts+2−0
containers/runner/src/api.ts+34−0
containers/runner/src/job.ts+16−1
containers/runner/test/job.test.ts+7−1
containers/runner/test/unit.test.ts+11−1
containers/runner/vitest.config.ts+4−0
packages/git-core/test/pack-limits.test.ts+16−10
packages/git-core/test/pack-unpack-stream.test.ts+17−10
packages/git-core/vitest.config.ts+4−0
packages/workflow-yaml/vitest.config.ts+4−0
vitest.shared.ts+10−0

addedapps/cli/vitest.config.ts+4−0View file

modifiedapps/diffinery/docs/actions/runners.md+1−1View file

modifiedapps/diffinery/src/repo/actions.ts+3−3View file

modifiedapps/diffinery/test/unit/actions-runtime.test.ts+1−1View file

modifiedapps/diffinery/vitest.config.ts+2−0View file

modifiedcontainers/runner/src/api.ts+34−0View file

modifiedcontainers/runner/src/job.ts+16−1View file

modifiedcontainers/runner/test/job.test.ts+7−1View file

modifiedcontainers/runner/test/unit.test.ts+11−1View file

addedcontainers/runner/vitest.config.ts+4−0View file

modifiedpackages/git-core/test/pack-limits.test.ts+16−10View file

modifiedpackages/git-core/test/pack-unpack-stream.test.ts+17−10View file

addedpackages/git-core/vitest.config.ts+4−0View file

addedpackages/workflow-yaml/vitest.config.ts+4−0View file

addedvitest.shared.ts+10−0View file

		@@ -1,0 +1,4 @@
	1	+import { defineConfig } from "vitest/config";
	2	+import { timeouts } from "../../vitest.shared";
	3	+
	4	+export default defineConfig({ test: timeouts });

		@@ -169,7 +169,7 @@
169	169
170	170	If the container stops unexpectedly (for example it runs out of memory), the job fails with `The runner stopped unexpectedly (exit code N).` A container that stops before its runner fetched the job is replaced instead, since no step ran.
171	171
172		-The log's first lines name where the job ran, for example `Runner: image diffinery-runner:96dbbe31790f, in fra06, EEUR, container 6f1c…`. The jobs API returns the same as `runner` (`image`, `location`, `instance`), next to `runner_name`. `GET /meta` names the image the last deploy set as `runner_image`. While a deploy's rollout is updating instances, a new container can still start on the previous image. Diffinery turns that runner away before it runs anything and starts the job again on another container (after 30 s, then 1, 2 and 4 minutes). After four tries, the job runs on whatever image it got rather than wait for a rollout that has stalled. The jobs API's `placementTries` counts the times a job went back in line before a runner took it, and `restarts` counts the times it started again because the platform replaced its runner.
	172	+The log's first lines name where the job ran, for example `Runner: image diffinery-runner:96dbbe31790f, in fra06, EEUR, container 6f1c…`. On hosted runners a `Host:` line names the host's processor (Cloudflare's hosts mix AMD EPYC generations, so the same job can run up to about twice as fast on one host as on another), and the line before the job's conclusion says how much of the vCPUs' time the host gave to other work (steal). The jobs API returns the same as `runner` (`image`, `location`, `instance`, and `cpu` on hosted runners), next to `runner_name`. `GET /meta` names the image the last deploy set as `runner_image`. While a deploy's rollout is updating instances, a new container can still start on the previous image. Diffinery turns that runner away before it runs anything and starts the job again on another container (after 30 s, then 1, 2 and 4 minutes). After four tries, the job runs on whatever image it got rather than wait for a rollout that has stalled. The jobs API's `placementTries` counts the times a job went back in line before a runner took it, and `restarts` counts the times it started again because the platform replaced its runner.
173	173
174	174	### Windows and macOS
175	175

		@@ -231,8 +231,8 @@
231	231	usage?: string \| null;
232	232	};
233	233
234		-/** Where a job ran, as its runner reported it: the runner image, the container's location and instance. */
235		-export type JobRunnerInfo = { image?: string; location?: string; instance?: string };
	234	+/** Where a job ran, as its runner reported it: the runner image, the container's location and instance, and the host's processor. */
	235	+export type JobRunnerInfo = { image?: string; location?: string; instance?: string; cpu?: string };
236	236
237	237	/** Reads the runner's `x-diffinery-runner` header (JSON), keeping only short string fields. */
238	238	export function parseRunnerInfo(header: string \| null \| undefined): JobRunnerInfo \| null {
		@@ -240,7 +240,7 @@
240	240	try {
241	241	const raw = JSON.parse(header) as Record<string, unknown>;
242	242	const out: JobRunnerInfo = {};
243		- for (const k of ["image", "location", "instance"] as const) if (typeof raw[k] === "string" && raw[k]) out[k] = (raw[k] as string).slice(0, 200);
	243	+ for (const k of ["image", "location", "instance", "cpu"] as const) if (typeof raw[k] === "string" && raw[k]) out[k] = (raw[k] as string).slice(0, 200);
244	244	return Object.keys(out).length ? out : null;
245	245	} catch {
246	246	return null;

		@@ -976,7 +976,7 @@
976	976
977	977	describe("runner info", () => {
978	978	it("keeps only the runner's short image, location and instance strings", () => {
979		- expect(parseRunnerInfo(JSON.stringify({ image: "diffinery-runner:abc", location: "fra06, EEUR", instance: "d0", token: "x", n: 1 }))).toEqual({ image: "diffinery-runner:abc", location: "fra06, EEUR", instance: "d0" });
	979	+ expect(parseRunnerInfo(JSON.stringify({ image: "diffinery-runner:abc", location: "fra06, EEUR", instance: "d0", cpu: "AMD EPYC (family 25, model 17), 4 vCPUs, 2550 MHz", token: "x", n: 1 }))).toEqual({ image: "diffinery-runner:abc", location: "fra06, EEUR", instance: "d0", cpu: "AMD EPYC (family 25, model 17), 4 vCPUs, 2550 MHz" });
980	980	expect(parseRunnerInfo(JSON.stringify({ image: "x".repeat(300) }))?.image).toHaveLength(200);
981	981	for (const bad of [null, "", "not json", "{}", JSON.stringify({ image: 5 }), "x".repeat(2000)]) expect(parseRunnerInfo(bad)).toBeNull();
982	982	});

		@@ -1,10 +1,12 @@
1	1	import { fileURLToPath } from "node:url";
2	2	import { defineConfig } from "vitest/config";
	3	+import { timeouts } from "../../vitest.shared";
3	4
4	5	// The path is base64-encoded so the id does not look like a node_modules file, which vitest would hand to Node unprocessed.
5	6	const PREFIX = "virtual:wasm-module/";
6	7
7	8	export default defineConfig({
	9	+ test: timeouts,
8	10	resolve: {
9	11	alias: {
10	12	"cloudflare:workflows": fileURLToPath(new URL("./test/unit/stubs/cloudflare-workflows.ts", import.meta.url)),

		@@ -127,9 +127,43 @@
127	127	const location = [env.CLOUDFLARE_LOCATION, env.CLOUDFLARE_REGION].filter(Boolean).join(", ");
128	128	if (location) out.location = location;
129	129	if (env.CLOUDFLARE_DURABLE_OBJECT_ID) out.instance = env.CLOUDFLARE_DURABLE_OBJECT_ID;
	130	+ const cpu = out.image ? hostCpu() : undefined;
	131	+ if (cpu) out.cpu = cpu;
130	132	return out;
131	133	}
132	134
	135	+const readText = (file: string): string => {
	136	+ try {
	137	+ return readFileSync(file, "utf8");
	138	+ } catch {
	139	+ return "";
	140	+ }
	141	+};
	142	+
	143	+/**
	144	+ * The host's processor as the container sees it. Cloudflare's hosts mix AMD EPYC generations whose
	145	+ * cores differ about 2x in speed (Zen 2, 3 and 4 were all seen on 2026-10-16), so a job's speed
	146	+ * depends on where it lands; the family and model say which one it got.
	147	+ */
	148	+export function hostCpu(cpuinfo = readText("/proc/cpuinfo")): string \| undefined {
	149	+ const field = (k: string) => new RegExp(`^${k}\\s:\\s(.*)$`, "m").exec(cpuinfo)?.[1]?.trim();
	150	+ const name = field("model name");
	151	+ if (!name) return undefined;
	152	+ const family = field("cpu family");
	153	+ const model = field("model");
	154	+ const mhz = Number(field("cpu MHz"));
	155	+ const vcpus = cpuinfo.match(/^processor\s*:/gm)?.length ?? 0;
	156	+ return [`${name}${family && model ? ` (family ${family}, model ${model})` : ""}`, vcpus && `${vcpus} vCPUs`, mhz > 0 && `${Math.round(mhz)} MHz`].filter(Boolean).join(", ");
	157	+}
	158	+
	159	+/** The vCPUs' time so far, from /proc/stat: all of it, and what the hypervisor gave to other work ("steal"). */
	160	+export function cpuTimes(stat = readText("/proc/stat")): { total: number; steal: number } \| null {
	161	+ const f = /^cpu\s+(.*)$/m.exec(stat)?.[1]?.trim().split(/\s+/).map(Number);
	162	+ if (!f \|\| f.length < 8 \|\| f.some((n) => !Number.isFinite(n))) return null;
	163	+ // user nice system idle iowait irq softirq steal (guest time is already in user).
	164	+ return { total: f.slice(0, 8).reduce((a, b) => a + b, 0), steal: f[7]! };
	165	+}
	166	+
133	167	function redactUrl(u: string): string {
134	168	return u.replace(/([?&](?:token\|sig\|signature)=)[^&]+/gi, "$1***");
135	169	}

		@@ -5,7 +5,7 @@
5	5	import path from "node:path";
6	6	import type { Conclusion, ContainerSpec, JobCompletion, JobSpec, StepReport, StepSpec } from "@diffinery/shared";
7	7	import { evaluate, evaluateIf, interpolate, type StatusFns } from "@diffinery/workflow-yaml";
8		-import { runnerInfo, type RunnerApi } from "./api.js";
	8	+import { cpuTimes, runnerInfo, type RunnerApi } from "./api.js";
9	9	import { parseKeyValueFile, parsePathFile } from "./commands.js";
10	10	import { Docker, type Wrap } from "./docker.js";
11	11	import { JobLogger, OutputProcessor, type Annotation, type OutputSink } from "./log.js";
		@@ -98,6 +98,8 @@
98	98	/** Set when the platform signals that it is replacing this runner (a container rollout). */
99	99	replacing = false;
100	100	private firstFailure: { step?: number; message: string } \| null = null;
	101	+ /** The vCPUs' time when a hosted job started, to say at the end how much of it the host took (steal). */
	102	+ private cpuAtStart: { total: number; steal: number } \| null = null;
101	103	private githubCtx: Record<string, unknown> = {};
102	104	private defaultEnv: Record<string, string> = {};
103	105	/**
		@@ -215,10 +217,19 @@
215	217	if (conclusion === "failure" && this.firstFailure) completion.failure = { ...this.firstFailure, message: this.masker.mask(this.firstFailure.message) };
216	218	if (conclusion === "cancelled") completion.failure = { message: "The job was cancelled." };
217	219	if (this.replacing && conclusion !== "success") completion.replaced = true;
	220	+ const stolen = this.stolenShare();
	221	+ if (stolen !== null) this.logger.line(-1, `Host: ${stolen}% of the vCPUs' time went to other work on the host (steal)`);
218	222	this.logger.line(-1, `Job completed with conclusion: ${conclusion}`);
219	223	return completion;
220	224	}
221	225
	226	+ /** The share of the vCPUs' time since the job started that the hypervisor gave to other work, in percent. */
	227	+ private stolenShare(): number \| null {
	228	+ const end = this.cpuAtStart && cpuTimes();
	229	+ if (!this.cpuAtStart \|\| !end \|\| end.total <= this.cpuAtStart.total) return null;
	230	+ return Math.round((100 * (end.steal - this.cpuAtStart.steal)) / (end.total - this.cpuAtStart.total));
	231	+ }
	232	+
222	233	// ------------------------------------------------------------ setup
223	234
224	235	private async setup(): Promise<void> {
		@@ -323,6 +334,10 @@
323	334	const where = runnerInfo();
324	335	if (Object.keys(where).length) this.logger.line(-1, `Runner: ${[where.image && `image ${where.image}`, where.location && `in ${where.location}`, where.instance && `container ${where.instance}`].filter(Boolean).join(", ")}`);
325	336	if (spec.runnerSize) this.logger.line(-1, spec.runnerSize);
	337	+ if (where.cpu) {
	338	+ this.logger.line(-1, `Host: ${where.cpu}`);
	339	+ this.cpuAtStart = cpuTimes();
	340	+ }
326	341	this.logger.line(-1, `Workspace: ${this.workspace}`);
327	342	// job-level env
328	343	const ctx = this.contexts({ steps: {}, inputs: spec.inputs ?? {}, failed: false, inheritedEnv: {}, composite: false, depth: 0 }, {});

		@@ -785,8 +785,14 @@
785	785	DIFFINERY_JOB_TOKEN: api.token,
786	786	});
787	787	expect(r2.status).toBe(1);
788		- expect(JSON.parse(api.requests.find((q) => q.path === "/runner/jobs/job-2")?.runner ?? "{}")).toEqual({ image: "diffinery-runner:abc123", location: "fra06, EEUR", instance: "d0" });
	788	+ const runner = JSON.parse(api.requests.find((q) => q.path === "/runner/jobs/job-2")?.runner ?? "{}");
	789	+ expect(runner).toMatchObject({ image: "diffinery-runner:abc123", location: "fra06, EEUR", instance: "d0" });
789	790	expect(api.logText()).toContain("Runner: image diffinery-runner:abc123, in fra06, EEUR, container d0");
	791	+ // A hosted runner also names its host's processor and, at the end, the share of its vCPUs' time the host took.
	792	+ if (existsSync("/proc/cpuinfo")) {
	793	+ expect(api.logText()).toContain(`Host: ${runner.cpu}`);
	794	+ expect(api.logText()).toMatch(/Host: \d+% of the vCPUs' time went to other work on the host \(steal\)/);
	795	+ }
790	796	expect(api.completion).toMatchObject({ conclusion: "failure", failure: { step: 0 } });
791	797	expect(readFileSync(path.join(workRoot, "w2/_temp/_github_workflow/event.json"), "utf8")).toContain("pusher");
792	798

		@@ -1,7 +1,7 @@
1	1	import { mkdirSync, writeFileSync } from "node:fs";
2	2	import path from "node:path";
3	3	import { describe, expect, it } from "vitest";
4		-import { LogBatcher } from "../src/api.js";
	4	+import { LogBatcher, cpuTimes, hostCpu } from "../src/api.js";
5	5	import { parseCommand, parseKeyValueFile, parsePathFile } from "../src/commands.js";
6	6	import { Masker } from "../src/mask.js";
7	7	import { parseUses, inputEnvName } from "../src/steps/actions.js";
		@@ -228,3 +228,13 @@
228	228	expect(received).toEqual(Array.from({ length: 10 }, (_, i) => `l${i}`));
229	229	});
230	230	});
	231	+
	232	+describe("host", () => {
	233	+ it("names the host's processor, and reads the vCPUs' stolen time", () => {
	234	+ const core = (n: number) => `processor\t: ${n}\nvendor_id\t: AuthenticAMD\ncpu family\t: 25\nmodel\t\t: 17\nmodel name\t: AMD EPYC\ncpu MHz\t\t: 2549.998\n\n`;
	235	+ expect(hostCpu([0, 1, 2, 3].map(core).join(""))).toBe("AMD EPYC (family 25, model 17), 4 vCPUs, 2550 MHz");
	236	+ expect(hostCpu("")).toBeUndefined();
	237	+ expect(cpuTimes("cpu 1498 0 227 267312 359 0 61 514 0 0\ncpu0 1 2 3 4 5 6 7 8 0 0\n")).toEqual({ total: 269971, steal: 514 });
	238	+ expect(cpuTimes("intr 1 2 3\n")).toBeNull();
	239	+ });
	240	+});

		@@ -51,16 +51,22 @@
51	51	const content = (i: number) => Buffer.from(String(i).padStart(8, "0"));
52	52	const oids = [await hashObject("blob", content(0))];
53	53	for (let i = 1; i <= n; i++) oids.push(await hashObject("blob", content(i)));
54		- const entries: Uint8Array[] = [];
55		- for (let i = n; i >= 1; i--) entries.push(refDelta(oids[i - 1]!, replace(8, content(i))));
56		- entries.push(blob(content(0)));
57		- const p = await pack(entries);
58		- const started = Date.now();
59		- const objects = await parsePack(p);
60		- expect(Date.now() - started).toBeLessThan(10_000);
61		- expect(objects).toHaveLength(n + 1);
62		- expect(new Set(objects.map((o) => o.oid))).toEqual(new Set(oids));
63		- }, 30_000);
	54	+ const deltas: Uint8Array[] = [];
	55	+ for (let i = 1; i <= n; i++) deltas.push(refDelta(oids[i - 1]!, replace(8, content(i))));
	56	+ const timed = async (entries: Uint8Array[]) => {
	57	+ const p = await pack(entries);
	58	+ const started = performance.now();
	59	+ const objects = await parsePack(p);
	60	+ return { ms: performance.now() - started, objects };
	61	+ };
	62	+ // Listed base first, every delta is resolvable when it is read: the time one pass takes on this machine.
	63	+ const onePass = await timed([blob(content(0)), ...deltas]);
	64	+ const newestFirst = await timed([...deltas.reverse(), blob(content(0))]);
	65	+ // Compared on the same machine, so a slower CI host doesn't change the verdict. Pass by pass, this took minutes.
	66	+ expect(newestFirst.ms).toBeLessThan(4 * onePass.ms + 500);
	67	+ expect(newestFirst.objects).toHaveLength(n + 1);
	68	+ expect(new Set(newestFirst.objects.map((o) => o.oid))).toEqual(new Set(oids));
	69	+ }, 120_000);
64	70
65	71	it("counts what each object costs in memory, not only its content", async () => {
66	72	// 20,000 one-byte blobs are a 240 KB pack but cost far more than 240 KB to hold.

		@@ -64,8 +64,8 @@
64	64	return new Uint8Array(Buffer.concat([body, createHash("sha1").update(body).digest()]));
65	65	}
66	66
67		-/** REF_DELTAs, each against the oid given, then one blob. */
68		-function refDeltaChain(deltas: { base: string; delta: Uint8Array }[], blob: Uint8Array): Uint8Array {
	67	+/** REF_DELTAs, each against the oid given, then one blob (or the blob first). */
	68	+function refDeltaChain(deltas: { base: string; delta: Uint8Array }[], blob: Uint8Array, blobFirst = false): Uint8Array {
69	69	const parts: Buffer[] = [Buffer.from("PACK"), Buffer.from([0, 0, 0, 2]), Buffer.alloc(4)];
70	70	parts[2]!.writeUInt32BE(deltas.length + 1);
71	71	const header = (code: number, size: number) => {
		@@ -80,8 +80,10 @@
80	80	head.push(c);
81	81	return Buffer.from(head);
82	82	};
	83	+ const blobPart = [header(3, blob.length), deflateSync(blob)];
	84	+ if (blobFirst) parts.push(...blobPart);
83	85	for (const d of deltas) parts.push(header(7, d.delta.length), Buffer.from(d.base, "hex"), deflateSync(d.delta));
84		- parts.push(header(3, blob.length), deflateSync(blob));
	86	+ if (!blobFirst) parts.push(...blobPart);
85	87	const body = Buffer.concat(parts);
86	88	return new Uint8Array(Buffer.concat([body, createHash("sha1").update(body).digest()]));
87	89	}
		@@ -229,7 +231,7 @@
229	231	await expect(parsePack(many, external, { maxBytes: base.length })).rejects.toThrow(/unpacks to more than/);
230	232	});
231	233
232		- it("resolves a long delta chain in one pass", { timeout: 60_000 }, async () => {
	234	+ it("resolves a long delta chain in one pass", { timeout: 120_000 }, async () => {
233	235	// Each delta's base is listed after it: resolving round by round took a round, and a pass over
234	236	// every delta left, per link.
235	237	const n = 20_000;
		@@ -239,12 +241,17 @@
239	241	const oids = await Promise.all(contents.map((c) => hashObject("blob", c)));
240	242	// Copy the 64 bytes the chain shares, then insert the two that number it.
241	243	const deltas = contents.slice(1).map((c, i) => ({ base: oids[i]!, delta: new Uint8Array([...varint(contents[i]!.length), ...varint(c.length), 0x90, 64, 2, c[64]!, c[65]!]) }));
242		- const pack = refDeltaChain(deltas.reverse(), blob);
243		- const t = Date.now();
244		- const seen = new Map<string, Uint8Array>();
245		- await StreamedPack.open(source(pack), {}, (o, content) => void seen.set(o.oid, content!));
246		- // Round by round, this took most of a minute.
247		- expect(Date.now() - t).toBeLessThan(20_000);
	244	+ const timed = async (pack: Uint8Array) => {
	245	+ const seen = new Map<string, Uint8Array>();
	246	+ const t = performance.now();
	247	+ await StreamedPack.open(source(pack), {}, (o, content) => void seen.set(o.oid, content!));
	248	+ return { ms: performance.now() - t, seen };
	249	+ };
	250	+ // Listed base first, every delta is resolvable when it is read: the time one pass takes on this machine.
	251	+ const onePass = await timed(refDeltaChain(deltas, blob, true));
	252	+ const { ms, seen } = await timed(refDeltaChain(deltas.reverse(), blob));
	253	+ // Compared on the same machine, so a slower CI host doesn't change the verdict. Round by round, this took most of a minute.
	254	+ expect(ms).toBeLessThan(4 * onePass.ms + 500);
248	255	expect(seen.size).toBe(n + 1);
249	256	expect(Buffer.from(seen.get(oids[n]!)!).equals(Buffer.from(contents[n]!))).toBe(true);
250	257	});

		@@ -1,0 +1,10 @@
	1	+/**
	2	+ * Unit-test settings every package shares.
	3	+ *
	4	+ * CI runs on Cloudflare's hosted runners, and Cloudflare places them on hosts whose cores differ
	5	+ * about 2x in speed (AMD EPYC Zen 2, 3 and 4 were all seen on 2026-10-16), some of them busy enough
	6	+ * to stall a container's vCPUs for seconds. A test's timeout only catches a hang, so it allows for
	7	+ * the slowest of those hosts; a test that checks speed says so itself, comparing against work done
	8	+ * on the same machine rather than against a fixed number of seconds.
	9	+ */
	10	+export const timeouts = { testTimeout: 30_000, hookTimeout: 30_000 };

codician/diffinerypublic

CI: test verdicts that don't depend on which host a runner lands on (#203)