Pushes: resolve deltas in one pass and bound what unpacking holds in memory (#162)

* Pushes: resolve deltas in one pass and bound what unpacking holds in memory

Found reviewing #157. parsePack resolved deltas by passing over the unresolved ones until none
were left, so a REF_DELTA chain listed newest first took one pass per link: 20,000 links took
minutes of the repository's Durable Object. And its only bound was 512 MB unpacked, which is not
a memory bound: under 200 KB of blobs just under 4 MiB, or a 240 KB pack of 20,000 tiny blobs at
about 740 bytes each to hold, ran the object out of its 128 MB.

- Deltas wait on their base (by offset or oid) and are applied as each base is added: one pass.
A thin pack's bases are read from the repository once each, and counted.
- maxHeldBytes (80 MiB by default) bounds what unpacking holds: contents under the large-blob
size, delta instructions until applied, a base while its deltas apply, and 768 bytes per
object. Each object is reserved at its declared size before inflating, and a delta's declared
result is checked against it before it is allocated. Past it the push is refused with
"split it into smaller pushes", which the importer already treats as too big and splits.
- A large blob kept compressed is inflated once for all the deltas against it, not once each.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo

* Deploy: check that /security/report shows Turnstile to people who aren't signed in

The deploy's sign-in page check now fetches /security/report anonymously and warns when it has
no Turnstile widget, so a deploy says whether people without an account can report (the sync
repeats the warning on GitHub).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo

---------

Co-authored-by: Claude <noreply@anthropic.com>

Canberk Demirci authored 12 days ago· GitHub committed 12 days agosigned1 parent 74272c2commit 1c15c4d14e12e4ac56d033a34e2134f6a59ff94cBrowse files

6 files changed+236−80

Files (6)

.github/workflows/deploy.yml+5−0
apps/diffinery/docs/reference/limits.md+1−0
apps/diffinery/docs/self-hosting/deploy.md+1−0
packages/git-core/src/pack.ts+109−75
packages/git-core/test/bombs.test.ts+5−5
packages/git-core/test/pack-limits.test.ts+115−0

modified.github/workflows/deploy.yml+5−0View file

modifiedapps/diffinery/docs/reference/limits.md+1−0View file

modifiedapps/diffinery/docs/self-hosting/deploy.md+1−0View file

modifiedpackages/git-core/src/pack.ts+109−75View file

modifiedpackages/git-core/test/bombs.test.ts+5−5View file

addedpackages/git-core/test/pack-limits.test.ts+115−0View file

		@@ -100,6 +100,11 @@
100	100	loc=$(curl -sS -o /dev/null -w '%{http_code} %{redirect_url}' https://$DIFFINERY_DOMAIN/auth/access)
101	101	echo "/auth/access -> $loc"
102	102	[[ "$loc" == 302\ https://.cloudflareaccess.com/ ]] \|\| echo "::warning::/auth/access is not behind Cloudflare Access yet, so email recovery and first admin sign-in will not work"
	103	+ if grep -q 'class="cf-turnstile"' <<<"$(curl -sS https://$DIFFINERY_DOMAIN/security/report)"; then
	104	+ echo "/security/report shows Turnstile to people who aren't signed in"
	105	+ else
	106	+ echo "::warning::/security/report has no Turnstile, so only people who are signed in can send security reports"
	107	+ fi
103	108	for kind in js css; do
104	109	asset=$(grep -o "/-/kumo/[a-z0-9]*\.$kind" <<<"$page" \| head -1)
105	110	[ -n "$asset" ] \|\| { echo "::error::/login does not load the Kumo $kind bundle"; exit 1; }

		@@ -135,6 +135,7 @@
135	135	\| Fetch request size \| 16 MB \| The request a client sends to fetch or clone (its wants and haves), after decompression. A larger one is refused with `413 fetch request exceeds 16 MB`. \|
136	136	\| File size in a push \| Up to the push size \| A single file (blob) can be as large as fits in one push (GitHub allows files up to 100 MB). Files of 4 MiB or more are checked and stored from their compressed bytes, so they never need to fit in memory inflated. \|
137	137	\| Unpacked size of a push \| 512 MiB \| What a push's pack inflates to, deltas resolved. A pack that unpacks to more is refused, however small it is on the wire. \|
	138	+\| Memory to unpack a push \| 80 MiB \| What unpacking a push holds at once: its files under 4 MiB inflated, and about 768 bytes for each object. A push that needs more (for example well over 100,000 objects, or many files under 4 MiB that compress well) is refused with `pack needs more than 80 MiB of memory to unpack; split it into smaller pushes`. Deltas are resolved in one pass, however the pack orders them. \|
138	139	\| Git protocol \| Version 2 for fetch and clone \| Clients older than git 2.26 must run `git config --global protocol.version 2`, or the server answers `Diffinery requires Git protocol v2`. \|
139	140	\| Secret scanning coverage \| Every new text file \| Push protection scans every new text file in a push, whatever its size. The rejection message lists up to 10 findings. \|
140	141	\| Linear history check \| 1,000 new commits \| A protected branch with "require linear history" checks up to 1,000 new commits in a push for merge commits. \|

		@@ -357,6 +357,7 @@
357	357	\| `https://.../healthz did not return 200 ... Check that the ... DNS record is proxied (orange cloud).` \| smoke test \| Check the DNS record (step 6) and the route in `wrangler.deploy.jsonc`: its pattern must be your hostname and its `zone_name` the zone that holds it (set `DIFFINERY_ZONE` if not). A new certificate can take a few minutes; rerun the workflow. \|
358	358	\| `/login is not the Diffinery sign-in page` \| smoke test \| Another service answers on the hostname. Check `DIFFINERY_DOMAIN`, `DIFFINERY_ZONE` and the route in `wrangler.deploy.jsonc`. \|
359	359	\| `/auth/access is not behind Cloudflare Access yet, so email recovery and first admin sign-in will not work` \| smoke test (warning) \| Check the Access step's output; the application must cover `<hostname>/auth/access`. \|
	360	+\| `/security/report has no Turnstile, so only people who are signed in can send security reports` \| smoke test (warning) \| The Turnstile step warned earlier in the deploy (see its message), or the operator's own `TURNSTILE_SITE_KEY` is set without `TURNSTILE_SECRET`. \|
360	361	\| `D1 queries don't run in a session (no x-d1-bookmark header)` \| smoke test \| The request to `/sitemap.xml` failed or the deployed Worker isn't the one you just built. Check that `wrangler deploy` succeeded and look at the Worker's logs. \|
361	362	\| "Account recovery by email is not set up on this instance. Ask an administrator for a sign-in link." \| `/auth/access` \| `ACCESS_TEAM_DOMAIN` or `ACCESS_AUD` is empty in the deployed config. Fix the Access step and redeploy. \|
362	363	\| "Cloudflare Access sign-in was rejected: audience mismatch" (or `issuer mismatch`) \| `/auth/access` \| The deployed `ACCESS_AUD` or `ACCESS_TEAM_DOMAIN` doesn't match the Access application. Rerun Deploy so bootstrap writes the current values. \|

		@@ -54,7 +54,7 @@
54	54	expect((await parsePack(p, undefined, { maxBytes: 2 * 1024 * 1024 })).length).toBe(2);
55	55	});
56	56
57		- it("counts every inflation of a large blob that tiny deltas use as their base", async () => {
	57	+ it("inflates a large blob that tiny deltas use as their base once for all of them, within the memory budget", async () => {
58	58	// One 2 MiB blob, kept compressed, then 20 one-byte deltas against it by offset.
59	59	const blob = Buffer.alloc(2 * 1024 * 1024);
60	60	const header = (type: number, size: number) => {
		@@ -85,10 +85,10 @@
85	85	const p = new Uint8Array(body.length + 20);
86	86	p.set(body);
87	87	p.set(await sha1(body), body.length);
88		- const opts = { maxBytes: 16 * 1024 * 1024, largeObjectBytes: 1024 * 1024 };
89		- await expect(parsePack(p, undefined, opts)).rejects.toThrow(/unpacks to more than/);
90		- // Kept whole, the blob is inflated once and the same pack fits.
91		- expect((await parsePack(p, undefined, { maxBytes: opts.maxBytes })).length).toBe(21);
	88	+ // Kept compressed, the blob is inflated once to hash it and once, held, for all 20 deltas: not once for each.
	89	+ const opts = { maxBytes: 5 * 1024 * 1024, largeObjectBytes: 1024 * 1024 };
	90	+ expect((await parsePack(p, undefined, opts)).length).toBe(21);
	91	+ await expect(parsePack(p, undefined, { ...opts, maxHeldBytes: 1024 * 1024 })).rejects.toThrow(/memory/);
92	92	});
93	93
94	94	it("checks a delta's result size before allocating it", () => {

		@@ -1,0 +1,115 @@
	1	+import { deflateSync, gzipSync } from "node:zlib";
	2	+import { describe, expect, it } from "vitest";
	3	+import { hashObject, parsePack, sha1 } from "../src/index.js";
	4	+
	5	+// Packs from clients must not run a repository's Durable Object out of memory (128 MB) or tie it up
	6	+// for minutes, however small the pack is.
	7	+
	8	+const header = (type: number, size: number) => {
	9	+ const out: number[] = [];
	10	+ let c = (type << 4) \| (size & 15);
	11	+ size = Math.floor(size / 16);
	12	+ while (size > 0) {
	13	+ out.push(c \| 0x80);
	14	+ c = size & 0x7f;
	15	+ size = Math.floor(size / 128);
	16	+ }
	17	+ out.push(c);
	18	+ return out;
	19	+};
	20	+const varint = (n: number) => {
	21	+ const out: number[] = [];
	22	+ do {
	23	+ let b = n % 128;
	24	+ n = Math.floor(n / 128);
	25	+ if (n > 0) b \|= 0x80;
	26	+ out.push(b);
	27	+ } while (n > 0);
	28	+ return out;
	29	+};
	30	+const hex = (oid: string) => Buffer.from(oid, "hex");
	31	+
	32	+/** A pack from entries already encoded (header, any base, zlib data). */
	33	+async function pack(entries: Uint8Array[]): Promise<Uint8Array> {
	34	+ const head = Buffer.alloc(12);
	35	+ head.write("PACK");
	36	+ head.writeUInt32BE(2, 4);
	37	+ head.writeUInt32BE(entries.length, 8);
	38	+ const body = Buffer.concat([head, ...entries]);
	39	+ return new Uint8Array(Buffer.concat([body, await sha1(body)]));
	40	+}
	41	+const blob = (content: Uint8Array, z = deflateSync) => Buffer.concat([Buffer.from(header(3, content.length)), z(content)]);
	42	+const refDelta = (base: string, delta: Uint8Array) => Buffer.concat([Buffer.from(header(7, delta.length)), hex(base), deflateSync(delta)]);
	43	+/** A delta that replaces a base of `from` bytes with `to`. */
	44	+const replace = (from: number, to: Uint8Array) => Uint8Array.from([...varint(from), ...varint(to.length), to.length, ...to]);
	45	+
	46	+describe("pack limits", () => {
	47	+ it("resolves a long delta chain in one pass, whatever order the pack lists it in", async () => {
	48	+ // 20,000 REF_DELTAs, each against the one before it, listed newest first: resolving them by
	49	+ // repeated passes over the unresolved ones took minutes.
	50	+ const n = 20_000;
	51	+ const content = (i: number) => Buffer.from(String(i).padStart(8, "0"));
	52	+ const oids = [await hashObject("blob", content(0))];
	53	+ for (let i = 1; i <= n; i++) oids.push(await hashObject("blob", content(i)));
	54	+ const entries: Uint8Array[] = [];
	55	+ for (let i = n; i >= 1; i--) entries.push(refDelta(oids[i - 1]!, replace(8, content(i))));
	56	+ entries.push(blob(content(0)));
	57	+ const p = await pack(entries);
	58	+ const started = Date.now();
	59	+ const objects = await parsePack(p);
	60	+ expect(Date.now() - started).toBeLessThan(10_000);
	61	+ expect(objects).toHaveLength(n + 1);
	62	+ expect(new Set(objects.map((o) => o.oid))).toEqual(new Set(oids));
	63	+ }, 30_000);
	64	+
	65	+ it("counts what each object costs in memory, not only its content", async () => {
	66	+ // 20,000 one-byte blobs are a 240 KB pack but cost far more than 240 KB to hold.
	67	+ const entries: Uint8Array[] = [];
	68	+ for (let i = 0; i < 20_000; i++) entries.push(blob(Buffer.from([i & 255, i >> 8])));
	69	+ const p = await pack(entries);
	70	+ await expect(parsePack(p, undefined, { maxHeldBytes: 1024 * 1024 })).rejects.toThrow(/memory/);
	71	+ expect(await parsePack(p, undefined, { maxHeldBytes: 64 * 1024 * 1024 })).toHaveLength(20_000);
	72	+ });
	73	+
	74	+ it("refuses a small pack whose objects inflate past the memory budget", async () => {
	75	+ // 25 blobs just under the 4 MiB that is kept compressed: under 200 KB of pack, about 100 MB held.
	76	+ const size = 4 * 1024 * 1024 - 1024;
	77	+ const entries: Uint8Array[] = [];
	78	+ for (let i = 0; i < 25; i++) {
	79	+ const b = Buffer.alloc(size);
	80	+ b[0] = i;
	81	+ entries.push(blob(b));
	82	+ }
	83	+ const p = await pack(entries);
	84	+ expect(p.length).toBeLessThan(250 * 1024);
	85	+ await expect(parsePack(p, undefined, { largeObjectBytes: 4 * 1024 * 1024 })).rejects.toThrow(/memory/);
	86	+ });
	87	+
	88	+ it("refuses a delta whose declared result can't be held, before allocating it", async () => {
	89	+ const base = Buffer.from("base");
	90	+ const delta = Uint8Array.from([...varint(base.length), ...varint(300 * 1024 * 1024), 4, ...base]);
	91	+ const p = await pack([blob(base), refDelta(await hashObject("blob", base), delta)]);
	92	+ await expect(parsePack(p)).rejects.toThrow(/delta result of \d+ bytes is larger/);
	93	+ });
	94	+
	95	+ it("counts a thin pack's bases from the repository, once each", async () => {
	96	+ const base = Buffer.alloc(2 * 1024 * 1024, 7);
	97	+ const oid = await hashObject("blob", base);
	98	+ let reads = 0;
	99	+ const external = async (o: string) => (o === oid ? (reads++, { type: "blob" as const, content: base }) : null);
	100	+ const deltas: Uint8Array[] = [];
	101	+ for (let i = 0; i < 3; i++) deltas.push(refDelta(oid, replace(base.length, Buffer.from(`v${i}`))));
	102	+ const p = await pack(deltas);
	103	+ await expect(parsePack(p, external, { maxBytes: 1024 * 1024 })).rejects.toThrow(/unpacks to more than/);
	104	+ reads = 0;
	105	+ expect(await parsePack(p, external, { maxBytes: 4 * 1024 * 1024 })).toHaveLength(3);
	106	+ expect(reads).toBe(1);
	107	+ });
	108	+
	109	+ it("refuses gzip data where git has zlib", async () => {
	110	+ const content = Buffer.from("hello\n");
	111	+ await expect(parsePack(await pack([blob(content, gzipSync)]))).rejects.toThrow(/zlib\|header/);
	112	+ const large = Buffer.alloc(64 * 1024, 1);
	113	+ await expect(parsePack(await pack([blob(large, gzipSync)]), undefined, { largeObjectBytes: 1024 })).rejects.toThrow(/zlib\|header/);
	114	+ });
	115	+});

		@@ -28,14 +28,10 @@
28	28	/** Packs bigger than this get their checksum computed in slices, so the pack is never copied for hashing. */
29	29	const SLICED_CHECKSUM_BYTES = 16 * 1024 * 1024;
30	30
31		-interface RawEntry {
	31	+/** A delta entry waiting for its base: its instructions (inflated), freed once applied. */
	32	+interface DeltaEntry {
32	33	offset: number;
33		- typeCode: number;
34		- data: Uint8Array; // inflated: full content, or delta instructions
35		- baseOffset?: number;
36		- baseOid?: string;
37		- /** A large blob that was hashed while inflating and not kept (see `largeObjectBytes`). */
38		- large?: PackedObject;
	34	+ data: Uint8Array;
39	35	}
40	36
41	37	export type ExternalLookup = (oid: string) => Promise<GitObject \| null>;
		@@ -44,12 +40,33 @@
44	40	export const MAX_UNPACKED_BYTES = 512 * 1024 * 1024;
45	41
46	42	/**
	43	+ * What unpacking a pack may hold in memory by default: the contents it returns (large blobs kept
	44	+ * compressed aside), delta instructions until they are applied, a base while its deltas are applied,
	45	+ * and OBJECT_OVERHEAD for each object. A repository's Durable Object has 128 MB, the pack included.
	46	+ */
	47	+export const MAX_HELD_BYTES = 80 * 1024 * 1024;
	48	+
	49	+/**
	50	+ * Roughly what one object costs to hold beyond its content: the object and its oid, the Uint8Array
	51	+ * and its buffer, and its entries in the indexes below. Measured at about 740 bytes in Node for
	52	+ * 100,000 three-byte blobs.
	53	+ */
	54	+const OBJECT_OVERHEAD = 768;
	55	+
	56	+const EMPTY = new Uint8Array(0);
	57	+
	58	+/**
47	59	* Parse a packfile and fully resolve every object (including deltas).
48	60	* `external` resolves REF_DELTA bases that are not inside the pack (thin packs).
49	61	* Packs come from clients, so unpacking is bounded: each object stops inflating at its
50		- * declared size, and the whole pack may unpack to at most `maxBytes`.
	62	+ * declared size, the whole pack may unpack to at most `maxBytes`, and what it holds in memory
	63	+ * may come to at most `maxHeldBytes`. Deltas are resolved in one pass from their bases.
51	64	*/
52		-export async function parsePack(pack: Uint8Array, external?: ExternalLookup, opts: { maxBytes?: number; largeObjectBytes?: number } = {}): Promise<PackedObject[]> {
	65	+export async function parsePack(
	66	+ pack: Uint8Array,
	67	+ external?: ExternalLookup,
	68	+ opts: { maxBytes?: number; largeObjectBytes?: number; maxHeldBytes?: number } = {},
	69	+): Promise<PackedObject[]> {
53	70	if (pack.length < 32 \|\| fromUtf8(pack.subarray(0, 4)) !== "PACK") throw new Error("not a packfile");
54	71	const view = new DataView(pack.buffer, pack.byteOffset, pack.byteLength);
55	72	const version = view.getUint32(4);
		@@ -66,11 +83,34 @@
66	83	if (!equalBytes(trailer, digest)) throw new Error("pack checksum mismatch");
67	84
68	85	const maxBytes = opts.maxBytes ?? MAX_UNPACKED_BYTES;
	86	+ const maxHeld = opts.maxHeldBytes ?? MAX_HELD_BYTES;
69	87	const large = opts.largeObjectBytes ?? Infinity;
70	88	let unpacked = 0;
71		- const tooBig = () => new Error(`pack unpacks to more than ${maxBytes} bytes`);
	89	+ let held = 0;
	90	+ const work = (n: number) => {
	91	+ if (unpacked + n > maxBytes) throw new Error(`pack unpacks to more than ${maxBytes} bytes`);
	92	+ unpacked += n;
	93	+ };
	94	+ const hold = (n: number) => {
	95	+ if (held + n > maxHeld) throw new Error(`pack needs more than ${Math.floor(maxHeld / (1024 * 1024))} MiB of memory to unpack; split it into smaller pushes`);
	96	+ held += n;
	97	+ };
72	98
73		- const raws: RawEntry[] = [];
	99	+ const results: PackedObject[] = [];
	100	+ /** Each result's offset in the pack, to find the OFS_DELTAs against it. */
	101	+ const offsets: number[] = [];
	102	+ const childrenByOffset = new Map<number, DeltaEntry[]>();
	103	+ const childrenByOid = new Map<string, DeltaEntry[]>();
	104	+ const waitFor = <K>(m: Map<K, DeltaEntry[]>, k: K, d: DeltaEntry) => {
	105	+ const list = m.get(k);
	106	+ if (list) list.push(d);
	107	+ else m.set(k, [d]);
	108	+ };
	109	+ const add = (offset: number, obj: PackedObject) => {
	110	+ results.push(obj);
	111	+ offsets.push(offset);
	112	+ };
	113	+
74	114	let pos = 12;
75	115	for (let n = 0; n < count; n++) {
76	116	const offset = pos;
		@@ -83,7 +123,8 @@
83	123	size += (c & 0x7f) * 2 ** shift;
84	124	shift += 7;
85	125	}
86		- const entry: RawEntry = { offset, typeCode, data: new Uint8Array(0) };
	126	+ let baseOffset: number \| undefined;
	127	+ let baseOid: string \| undefined;
87	128	if (typeCode === OFS_DELTA) {
88	129	c = pack[pos++]!;
89	130	let ofs = c & 0x7f;
		@@ -91,91 +132,84 @@
91	132	c = pack[pos++]!;
92	133	ofs = (ofs + 1) * 128 + (c & 0x7f);
93	134	}
94		- entry.baseOffset = offset - ofs;
	135	+ baseOffset = offset - ofs;
95	136	} else if (typeCode === REF_DELTA) {
96		- entry.baseOid = toHex(pack.subarray(pos, pos + 20));
	137	+ baseOid = toHex(pack.subarray(pos, pos + 20));
97	138	pos += 20;
98	139	} else if (!CODE_TYPES[typeCode]) {
99	140	throw new Error(`bad object type ${typeCode} at ${offset}`);
100	141	}
101		- if (unpacked + size > maxBytes) throw tooBig();
102	142	if (typeCode === TYPE_CODES.blob && size >= large) {
103	143	// Hash it as it inflates and keep only the compressed bytes, so memory never holds it whole.
	144	+ work(size);
	145	+ hold(OBJECT_OVERHEAD);
104	146	const h = new Sha1().update(objectHeader("blob", size));
105	147	const { consumed, produced } = inflateEach(pack, pos, (chunk) => h.update(chunk), size);
106	148	if (produced !== size) throw new Error(`size mismatch at ${offset}: ${produced} != ${size}`);
107		- entry.large = largeBlob(toHex(h.digest()), size, pack.subarray(pos, pos + consumed));
108		- unpacked += size;
	149	+ add(offset, largeBlob(toHex(h.digest()), size, pack.subarray(pos, pos + consumed)));
109	150	pos += consumed;
110		- raws.push(entry);
111	151	continue;
112	152	}
	153	+ // Reserved at the declared size before inflating, so a declared size the budget can't hold is refused up front.
	154	+ work(size);
	155	+ hold(size + OBJECT_OVERHEAD);
113	156	const { out, consumed } = inflateAt(pack, pos, size);
114	157	if (out.length !== size) throw new Error(`size mismatch at ${offset}: ${out.length} != ${size}`);
115		- unpacked += size;
116		- entry.data = out;
117	158	pos += consumed;
118		- raws.push(entry);
	159	+ if (baseOffset !== undefined) waitFor(childrenByOffset, baseOffset, { offset, data: out });
	160	+ else if (baseOid !== undefined) waitFor(childrenByOid, baseOid, { offset, data: out });
	161	+ else {
	162	+ const type = CODE_TYPES[typeCode]!;
	163	+ add(offset, { oid: await hashObject(type, out), type, content: out });
	164	+ }
119	165	}
120	166	if (pos !== pack.length - 20) throw new Error("trailing garbage in pack");
121	167
122		- const byOffset = new Map<number, PackedObject>();
123		- const byOid = new Map<string, PackedObject>();
124		- const results: PackedObject[] = [];
125		- const add = async (raw: RawEntry, type: ObjectType, content: Uint8Array) => {
126		- const obj: PackedObject = { oid: await hashObject(type, content), type, content };
127		- byOffset.set(raw.offset, obj);
128		- byOid.set(obj.oid, obj);
129		- results.push(obj);
130		- };
131		-
132		- let pending: RawEntry[] = [];
133		- for (const raw of raws) {
134		- const t = CODE_TYPES[raw.typeCode];
135		- if (raw.large) {
136		- byOffset.set(raw.offset, raw.large);
137		- byOid.set(raw.large.oid, raw.large);
138		- results.push(raw.large);
139		- } else if (t) await add(raw, t, raw.data);
140		- else pending.push(raw);
141		- }
142		-
143		- // Resolve deltas; chains may reference each other in any order.
144		- while (pending.length) {
145		- const next: RawEntry[] = [];
146		- for (const raw of pending) {
147		- let base: GitObject \| undefined \| null;
148		- if (raw.baseOffset !== undefined) base = byOffset.get(raw.baseOffset);
149		- else base = byOid.get(raw.baseOid!);
150		- if (!base) {
151		- next.push(raw);
152		- continue;
153		- }
154		- // A large blob kept compressed is inflated again for every delta against it. Count that work
155		- // (or the result, if larger), so many tiny deltas against one large blob can't make the server
156		- // inflate it without bound.
157		- const large = (base as PackedObject).deflated ? ((base as PackedObject).size ?? 0) : 0;
158		- if (unpacked + large > maxBytes) throw tooBig();
159		- const content = applyDelta(base.content, raw.data, maxBytes - unpacked);
160		- unpacked += Math.max(content.length, large);
161		- await add(raw, base.type, content);
	168	+ /** Applies every delta waiting for `base` (in the pack at `offset`, or -1 when it came from the repository). */
	169	+ const resolveDeltasOf = async (base: GitObject & { oid: string; deflated?: Uint8Array; size?: number }, offset: number) => {
	170	+ const byOffset = offset >= 0 ? childrenByOffset.get(offset) : undefined;
	171	+ const byOid = childrenByOid.get(base.oid);
	172	+ if (!byOffset && !byOid) return;
	173	+ if (byOffset) childrenByOffset.delete(offset);
	174	+ if (byOid) childrenByOid.delete(base.oid);
	175	+ // A large blob kept compressed is inflated once for all its deltas, and held while they are applied.
	176	+ const inflated = base.deflated ? (base.size ?? 0) : 0;
	177	+ if (inflated) {
	178	+ work(inflated);
	179	+ hold(inflated);
162	180	}
163		- if (next.length === pending.length) {
164		- // No progress: remaining bases must come from outside the pack.
165		- let progressed = false;
166		- for (const raw of next) {
167		- if (raw.baseOid && external) {
168		- const base = await external(raw.baseOid);
169		- if (base) {
170		- byOid.set(raw.baseOid, { oid: raw.baseOid, ...base });
171		- progressed = true;
172		- }
173		- }
174		- }
175		- if (!progressed) throw new Error(`unresolvable delta base(s): ${next.map((r) => r.baseOid ?? `@${r.baseOffset}`).join(", ")}`);
	181	+ const content = base.content;
	182	+ for (const d of [...(byOffset ?? []), ...(byOid ?? [])]) {
	183	+ const result = applyDelta(content, d.data, Math.min(maxBytes - unpacked, maxHeld - held + d.data.length));
	184	+ work(result.length);
	185	+ held -= d.data.length;
	186	+ hold(result.length);
	187	+ d.data = EMPTY;
	188	+ add(d.offset, { oid: await hashObject(base.type, result), type: base.type, content: result });
176	189	}
177		- pending = next.filter((r) => !byOffset.has(r.offset));
	190	+ held -= inflated;
	191	+ };
	192	+ // Each object, as it is added, has its deltas applied: one pass, whatever order the pack lists them in.
	193	+ let next = 0;
	194	+ const drain = async () => {
	195	+ for (; next < results.length; next++) await resolveDeltasOf(results[next]!, offsets[next]!);
	196	+ };
	197	+ await drain();
	198	+ // REF_DELTA bases that aren't in the pack (thin packs) come from the repository, each read once.
	199	+ if (external) {
	200	+ for (const oid of [...childrenByOid.keys()]) {
	201	+ if (!childrenByOid.has(oid)) continue;
	202	+ const base = await external(oid);
	203	+ if (!base) continue;
	204	+ work(base.content.length);
	205	+ hold(base.content.length);
	206	+ await resolveDeltasOf({ oid, ...base }, -1);
	207	+ held -= base.content.length;
	208	+ await drain();
	209	+ }
178	210	}
	211	+ const unresolved = [...childrenByOid.keys(), ...[...childrenByOffset.keys()].map((o) => `@${o}`)];
	212	+ if (unresolved.length) throw new Error(`unresolvable delta base(s): ${unresolved.join(", ")}`);
179	213	return results;
180	214	}
181	215

codician/diffinerypublic

Pushes: resolve deltas in one pass and bound what unpacking holds in memory (#162)