Pushes: refuse a commit or tag over 4 MiB (#175)

Found reviewing #157. fsck capped trees but not commits or tags: a pushed commit with a 40 MiB
message passed, and after the push readCommit decoded it whole (workflow triggers, webhooks), about
80 MiB of heap on top of the 40 MiB object, enough to run the repository's Durable Object out of
memory. Both push paths now refuse a commit or tag over 4 MiB in fsck; real ones are a few KB.


Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo

Co-authored-by: Claude <noreply@anthropic.com>

Canberk Demirci authored 12 days ago· GitHub committed 12 days agosigned1 parent 1ccd94bcommit 3281dab50483d091e01093308139ceec26e4d170Browse files

5 files changed+30−9

Files (5)

apps/diffinery/docs/reference/limits.md+1−0
apps/diffinery/docs/repositories/git.md+1−1
apps/diffinery/src/git/fsck.ts+14−2
apps/diffinery/src/git/push-contents.ts+4−5
apps/diffinery/test/unit/fsck.test.ts+10−1

modifiedapps/diffinery/docs/reference/limits.md+1−0View file

modifiedapps/diffinery/docs/repositories/git.md+1−1View file

modifiedapps/diffinery/src/git/fsck.ts+14−2View file

modifiedapps/diffinery/src/git/push-contents.ts+4−5View file

modifiedapps/diffinery/test/unit/fsck.test.ts+10−1View file

		@@ -136,6 +136,7 @@
136	136	\| File size in a push \| Up to the push size \| A single file (blob) can be as large as fits in one push (GitHub allows files up to 100 MB). Files of 4 MiB or more are checked and stored from their compressed bytes, so they never need to fit in memory inflated. \|
137	137	\| Unpacked size of a push \| 512 MiB \| What a push's pack inflates to, deltas resolved. A pack that unpacks to more is refused with `pack unpacks to more than 512 MiB; split it into smaller pushes`, however small it is on the wire: a long history of one large file can, since each version counts whole. \|
138	138	\| Tree size in a push \| 8 MiB \| One directory's listing (a tree object). A larger one is refused with `fsck error: tree <id>: larger than 8 MiB`. 100,000 files in one directory come to about 4 MiB. \|
	139	+\| Commit or tag size in a push \| 4 MiB \| A commit or annotated tag object, message and signature included. A larger one is refused with `fsck error: commit <id>: larger than 4 MiB` (or `tag`). \|
139	140	\| Objects in a push unpacked in pieces \| 1,000,000 \| A push over 16 MB is unpacked from storage in pieces, and so is a smaller one whose objects don't all fit in memory at once (a long history, such as flask's or tornado's). Inflating its objects again to check and store them may take at most four times the unpacked size limit, and reading it from storage at most 500 requests, counting bases read from the repository and each object stored on its own (jQuery's 37 MB history takes about 50). Past any of these, it is refused with a message that ends `split it into smaller pushes`. Its deltas count toward the storage quota at their full size. \|
140	141	\| Memory to unpack a push \| 80 MiB \| What unpacking a push holds at once. Up to 16 MB: its files under 4 MiB inflated, and about 768 bytes for each object. Unpacked in pieces: 24 MiB of caches and scratch, up to 16 MiB of deltas kept to rebuild them, each object while it is in use (a file, a delta's base and its result, a base read from the repository), about 128 bytes for each object, and 640 more for each delta named by its base's id rather than its place in the pack (git sends those only for bases the repository already has). Two versions of a file over about 25 MB, which git stores as a delta, need more; a long history of one file holds two versions at a time. A base from the repository is checked before it is read. A push that needs more is refused with `pack needs more than 80 MiB of memory to unpack; split it into smaller pushes`. Deltas are resolved in one pass, however the pack orders them. \|
141	142	\| Git protocol \| Version 2 for fetch and clone \| Clients older than git 2.26 must run `git config --global protocol.version 2`, or the server answers `Diffinery requires Git protocol v2`. \|

		@@ -340,7 +340,7 @@
340	340	\| `remote unpack failed: the push needs more memory than one push may use; split it into smaller pushes (...)` \| Unpacking or storing this push ran out of the memory one push may use, even though it is under the size limits. Pushes over 16 MB are unpacked in pieces, so this now comes from a single very large file, or a delta against one. Nothing from the push is kept. \| Push history in steps, oldest first, as for the 100 MB limit below. The GitHub importer does this by itself. \|
341	341	\| `remote unpack failed: pack needs more than 80 MiB of memory to unpack; split it into smaller pushes` \| Unpacking would hold more than 80 MiB at once: files under 4 MiB inflated, plus about 768 bytes per object. Very many objects (well over 100,000) or many small files that compress well hit it before the size limits. A push unpacked in pieces (over 16 MB, or one that didn't fit) counts its caches, the files in use and about 128 bytes per object instead, so there it comes from very large files. See [Limits](/docs/reference/limits). \| Push history in steps, oldest first, as for the 100 MB limit below. The GitHub importer does this by itself. \|
342	342	\| `remote unpack failed: push exceeds 100 MB; split it into smaller pushes` \| The pack for this push is larger than 100 MB. \| Push history in steps, oldest first, for example `git push origin <older-commit>:refs/heads/main`, then later commits. Move large binaries to Git LFS. \|
343		-\| `remote: Push refused: tree <id>: duplicate entry '<name>'` (or another reason), `remote unpack failed: fsck error: ...` and `[remote rejected] ... (fsck failed)` \| A tree or `.gitmodules` file in the push fails git's `fsck` checks: an entry that is empty, `.`, `..`, a spelling of `.git`, contains `/` or appears twice, a tree over 8 MiB, a `.gitmodules` that is a symbolic link, or a submodule whose name has `..` or whose `url` or `path` starts with `-`. Nothing from the push is kept. \| Run `git fsck` locally to find the object, fix the tree or `.gitmodules` in a new commit (or rewrite the commit that added it), and push again. \|
	343	+\| `remote: Push refused: tree <id>: duplicate entry '<name>'` (or another reason), `remote unpack failed: fsck error: ...` and `[remote rejected] ... (fsck failed)` \| A tree or `.gitmodules` file in the push fails git's `fsck` checks: an entry that is empty, `.`, `..`, a spelling of `.git`, contains `/` or appears twice, a tree over 8 MiB, a commit or tag over 4 MiB, a `.gitmodules` that is a symbolic link, or a submodule whose name has `..` or whose `url` or `path` starts with `-`. Nothing from the push is kept. \| Run `git fsck` locally to find the object, fix the tree or `.gitmodules` in a new commit (or rewrite the commit that added it), and push again. \|
344	344	\| `[remote rejected] ... (protected branch: required check '<name>' has not run on <sha>; ...)` \| The branch requires status checks, and the commit you pushed hasn't passed them. \| Push to another branch and open a pull request, or push a commit that already passed the checks. \|
345	345	\| `[remote rejected] ... (secret detected (push protection))` \| A pushed file contains something that looks like a credential. \| Remove it from every commit in the push (`git commit --amend`, or `git rebase -i`), rotate the credential, and push again. See [Push protection](/docs/security/push-protection). \|
346	346	\| `[remote rejected] ... (fetch first (stale info))` \| The branch moved since you fetched. \| `git pull --rebase` (or fetch and merge), then push. \|

		@@ -53,8 +53,7 @@
53	53	export function fsckObjects(objects: PushedObject[]): string \| null {
54	54	const gitmodules = new Set<string>();
55	55	for (const o of objects) {
56		- if (o.type !== "tree") continue;
57		- const why = fsckTree(o.oid, o.content, gitmodules);
	56	+ const why = o.type === "tree" ? fsckTree(o.oid, o.content, gitmodules) : fsckSize(o.oid, o.type, o.content.length);
58	57	if (why) return why;
59	58	}
60	59	if (gitmodules.size) {
		@@ -73,6 +72,19 @@
73	72	*/
74	73	export const MAX_TREE_BYTES = 8 * 1024 * 1024;
75	74
	75	+/**
	76	+ * The largest commit or tag a push may hold. After a push, its commits are read and decoded whole
	77	+ * (workflow triggers, webhooks, the pull request page), at about three times their size; a real
	78	+ * commit message or tag is a few KB.
	79	+ */
	80	+export const MAX_COMMIT_BYTES = 4 * 1024 * 1024;
	81	+
	82	+/** Why a pushed commit or tag is too large to accept, or null (any other type passes). */
	83	+export function fsckSize(oid: string, type: string, size: number): string \| null {
	84	+ if ((type === "commit" \|\| type === "tag") && size > MAX_COMMIT_BYTES) return `${type} ${oid}: larger than ${MAX_COMMIT_BYTES / (1024 * 1024)} MiB`;
	85	+ return null;
	86	+}
	87	+
76	88	/** Why one pushed tree can't be accepted, or null. The blobs it names `.gitmodules` go into `gitmodules`, for `fsckGitmodules`. */
77	89	export function fsckTree(oid: string, content: Uint8Array, gitmodules: Set<string>): string \| null {
78	90	if (content.length > MAX_TREE_BYTES) return `tree ${oid}: larger than ${MAX_TREE_BYTES / (1024 * 1024)} MiB`;

		@@ -1,6 +1,6 @@
1	1	import { StreamedPack, type ExternalLookup, type ObjectType, type PackSource, type PackedObject } from "@diffinery/git-core";
2	2	import { INLINE_LIMIT, type RepoObjectStore } from "../repo/objects.js";
3		-import { fsckGitmodules, fsckObjects, fsckTree } from "./fsck.js";
	3	+import { fsckGitmodules, fsckObjects, fsckSize, fsckTree } from "./fsck.js";
4	4	import { parseReceivePackHead, type ReceiveRequest } from "./protocol.js";
5	5	import { scanFeed, scanForSecrets, type SecretHit } from "./secret-scan.js";
6	6
		@@ -100,10 +100,9 @@
100	100	let pack: StreamedPack \| null = null;
101	101	try {
102	102	const p = (pack = await StreamedPack.open(src, { external: repositoryBases(store), largeObjectBytes: o.largeObjectBytes, writesOver: INLINE_LIMIT }, async (obj, content, feed) => {
103		- if (obj.type === "tree") {
104		- const why = fsckTree(obj.oid, content!, gitmodules);
105		- if (why) throw new FsckRefusal(why);
106		- } else if (scanned(obj.type)) {
	103	+ const why = obj.type === "tree" ? fsckTree(obj.oid, content!, gitmodules) : content ? fsckSize(obj.oid, obj.type, content.length) : null;
	104	+ if (why) throw new FsckRefusal(why);
	105	+ if (scanned(obj.type)) {
107	106	if (content) scanForSecrets([{ oid: obj.oid, content }], opted).forEach((h) => hits.push(h));
108	107	else await scanFeed(obj.oid, feed!, hits, opted);
109	108	}

		@@ -1,5 +1,5 @@
1	1	import { describe, expect, it } from "vitest";
2		-import { MAX_TREE_BYTES, fsckObjects, isDotGitmodulesName, unsafeGitmodules } from "../../src/git/fsck.js";
	2	+import { MAX_COMMIT_BYTES, MAX_TREE_BYTES, fsckObjects, fsckSize, isDotGitmodulesName, unsafeGitmodules } from "../../src/git/fsck.js";
3	3
4	4	const enc = new TextEncoder();
5	5	const OID = "a".repeat(40);
		@@ -72,4 +72,13 @@
72	72	expect(fsckObjects([big])).toBeNull();
73	73	expect(fsckObjects([{ ...big, content: big.content.subarray(0, big.content.length - 3) }])).toMatch(/malformed tree/);
74	74	});
	75	+
	76	+ it("refuses a commit or tag over 4 MiB, which would be decoded whole after the push", () => {
	77	+ const big = (type: string, n: number) => ({ oid: "c".repeat(40), type, content: new Uint8Array(n).fill(0x61) });
	78	+ expect(fsckObjects([big("commit", MAX_COMMIT_BYTES)])).toBeNull();
	79	+ expect(fsckObjects([big("commit", MAX_COMMIT_BYTES + 1)])).toBe(`commit ${"c".repeat(40)}: larger than 4 MiB`);
	80	+ expect(fsckObjects([big("tag", MAX_COMMIT_BYTES + 1)])).toMatch(/^tag c+: larger than 4 MiB$/);
	81	+ // Blobs of any size are for the large-object path, not this check.
	82	+ expect(fsckSize("b".repeat(40), "blob", 100 * 1024 * 1024)).toBeNull();
	83	+ });
75	84	});

codician/diffinerypublic

Pushes: refuse a commit or tag over 4 MiB (#175)