Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Repository admins can purge objects no ref reaches, so a secret force-pushed away can't be fetched by its SHA (#93)

Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo

Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent de9b88fcommit 3675d7a4162ca01314f5704e25848ba6a835ea27Browse files
9 files changed+163−2
Files (9)
modifiedapps/diffinery/docs/reference/limits.md+1−0View file
@@ -183,6 +183,7 @@
183183 | Hosted jobs at once per repository | 20 | Further jobs stay queued and start, oldest first, as the repository's own jobs finish, so one repository can't take every container. Self-hosted jobs don't count. |
184184 | Unfinished runs per repository | 500 | Past it, a `workflow_dispatch` or `repository_dispatch` is refused with `429 this repository has 500 unfinished workflow runs, the most allowed; wait for some to finish`, and other events start no run. |
185185 | Self-hosted runners per repository | 100, each with at most 20 labels | Registering one more gets `429 this repository has 100 self-hosted runners, the most allowed; remove one first`. A registration token works for an hour. |
186+| Purging unreachable objects | Repositories of up to 300,000 objects | A larger repository gets `422`; ask a site administrator. See [Removing a secret that was pushed](/docs/security/push-protection#removing-a-secret-that-was-pushed). |
186187 | Jobs during a deploy | Not interrupted | A deploy doesn't replace the container of a running hosted job: it finishes on the version it started with. If the platform replaces a job's container anyway, the job keeps going for up to 13 minutes, then is cancelled. |
187188 | Service container health check | 15 minutes | How long a job waits for a `services:` container's health check to pass. |
188189 | Log line length | 16,384 characters | Longer lines are cut and end with `…`. |
modifiedapps/diffinery/docs/security/audit-log.md+1−0View file
@@ -107,6 +107,7 @@
107107 | `repo.update` | **General** settings are saved, or `PATCH /api/v3/repos/{owner}/{repo}` changes the name, description, visibility or default branch | Web: `name`, `visibility`, `defaultBranch`. API: the fields that were sent (`name`, `description`, `visibility`, `default_branch`) |
108108 | `repo.archive`, `repo.unarchive` | The repository is archived or unarchived, on **General** or with `PATCH /api/v3/repos/{owner}/{repo}` and `{"archived": true}` (or `false`) | none |
109109 | `repo.delete` | The repository is deleted (web or `DELETE /api/v3/repos/{owner}/{repo}`) | none |
110+| `repo.purge_unreachable` | Unreachable objects are purged (**Settings → Danger zone**, or `POST /api/v3/repos/{owner}/{repo}/diffinery/purge-unreachable`) | `objects`, `bytes` |
110111 | `repo.transfer` | The repository is transferred to another owner (web or `POST /api/v3/repos/{owner}/{repo}/transfer`); the target is the new name | `from`, `to` |
111112 | `repo.collaborator.set`, `repo.collaborator.remove` | On **Collaborators**: a person or agent with an account is added with **Add people**, or a collaborator's role is changed, or a collaborator is removed. Through the API: `PUT` or `DELETE /api/v3/repos/{owner}/{repo}/collaborators/{username}` | `login`, `role` |
112113 | `repo.invite` | An email address without an account is entered in **Add people**, which sends an invitation | `email`, `role` |
modifiedapps/diffinery/docs/security/overview.md+1−1View file
@@ -61,7 +61,7 @@
6161 - creating a personal access token, an agent, or an agent token;
6262 - changing your username;
6363 - issuing a sign-in link (site administrators);
64-- deleting or transferring a repository;
64+- deleting or transferring a repository, or purging its unreachable objects;
6565 - deleting an organization.
6666
6767 When a browser's sign-in is older than that, Diffinery asks you to **Confirm it's you** with your passkey (or by email recovery, where that is set up), then takes you back to the page. Then submit the form again. The same calls through the API with a browser session get `403 Confirm it's you: sign in with your passkey again to do this.`, with a `confirm` URL in the body. So a session left open on a shared computer, or a stolen session cookie, can't add its own passkey, mint a lasting token or delete your work. Tokens aren't affected: their scopes bound what they can do.
modifiedapps/diffinery/docs/security/push-protection.md+10−0View file
@@ -219,6 +219,16 @@
219219
220220 Bypassing does not make a real credential safe: anyone who can read the repository can read it. Rotate real credentials even after a bypass.
221221
222+## Removing a secret that was pushed
223+
224+Force-pushing a branch to drop a commit doesn't delete the commit. Like git, Diffinery keeps objects no branch reaches, and anyone who can read the repository can still fetch the commit by its SHA (`GET .../git/commits/{sha}`, the commit page, or `git fetch origin <sha>`). To remove it:
225+
226+1. **Rotate the credential** at its provider. Treat it as exposed: clones, forks, caches and webhook receivers may already have it.
227+2. Rewrite the history without it and force-push every branch and tag that contains it. Close or update any pull request whose head contains it.
228+3. On the repository's **Settings**, under **Danger zone**, click **Purge unreachable objects** (or `POST /api/v3/repos/{owner}/{repo}/diffinery/purge-unreachable`). It permanently deletes every commit, tree and file that no branch, tag, pull request, merge queue entry or unfinished workflow run reaches, and empties the cache of downloadable archives. It answers `{"objects": 3, "bytes": 1234}`.
229+
230+Only repository admins (people, not agents) can purge, with a sign-in from the last 2 hours in the browser. It is recorded in the [audit log](/docs/security/audit-log) as `repo.purge_unreachable`. A push that was in flight while the purge ran is refused with `the repository's unreachable objects were purged during this push; push again`. Purging works for repositories of up to 300,000 objects. Review comments on commits that were purged no longer show their diff.
231+
222232 ## Imported history
223233
224234 An [import from GitHub](/docs/migrate-from-github) copies a repository's Git history by pushing it into Diffinery in chunks, and push protection scans every one of those pushes like any other: each new text blob, with the same patterns and the same `diffinery:allow-secret` opt-out. Git LFS objects and release assets are not scanned, as for any push. Old history often holds credentials that were committed and removed later, so every import has a setting for what happens when push protection finds one:
modifiedapps/diffinery/src/http/api.ts+8−0View file
@@ -261,6 +261,14 @@
261261 if (Object.keys(changed).some((k) => k !== "archived")) auditRepo(c, caller, repo, "repo.update", changed);
262262 return c.json(repoOut(c, updated, caller.role));
263263 });
264+// Delete every object no ref, open pull request, merge queue entry or unfinished run reaches (after force-pushing a secret away).
265+api.post("/repos/:owner/:repo/diffinery/purge-unreachable", async (c) => {
266+ const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin");
267+ requireRecentSignIn(c, `/${repo.owner_login}/${repo.name}/settings`);
268+ const r = await stub.purgeUnreachable(caller);
269+ auditRepo(c, caller, repo, "repo.purge_unreachable", { objects: r.objects, bytes: r.bytes, via: "api" });
270+ return c.json({ objects: r.objects, bytes: r.bytes });
271+});
264272 api.delete("/repos/:owner/:repo", async (c) => {
265273 const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin", { whileArchived: true });
266274 if (caller.kind === "agent") throw new HttpError(403, "agents cannot delete repositories");
modifiedapps/diffinery/src/repo/RepoDO.ts+72−1View file
@@ -3,6 +3,7 @@
33 import { REVIEW_APP, REVIEW_CHECK } from "../agentic/review.js";
44 import { unsafeTreePath } from "../lib/tree-names.js";
55 import {
6+ CODE_TYPES,
67 EMPTY_TREE_OID,
78 ZERO_OID,
89 applyChanges,
@@ -16,7 +17,9 @@
1617 isOid,
1718 isTreeMode,
1819 mergeBase,
20+ parseCommit,
1921 parsePack,
22+ parseTree,
2023 peelToCommit,
2124 readCommit,
2225 readTree,
@@ -43,7 +46,7 @@
4346 import { fsckObjects } from "../git/fsck.js";
4447 import { scanForSecrets, secretBypassReason, type SecretHit } from "../git/secret-scan.js";
4548 import { HttpError, isValidRefName, now } from "../lib/util.js";
46-import { deletePrefixes, repoPrefixes } from "../lib/storage-keys.js";
49+import { archivePrefix, deletePrefixes, repoPrefixes } from "../lib/storage-keys.js";
4750 import { Actions } from "./actions.js";
4851 import { AutoMerge, isPerson } from "./auto-merge.js";
4952 import { Collab } from "./collab.js";
@@ -64,6 +67,8 @@
6467 import { atLeast, globToRegExp, type Caller, type ProtectionRules, type RepoMeta } from "./types.js";
6568
6669 const MAX_PUSH_BYTES = 100 * 1024 * 1024;
70+/** A purge walks every reachable commit and tree while holding the repository: bounded so it finishes in time. */
71+export const MAX_PURGE_OBJECTS = 300_000;
6772 /** Storage key that marks a deleted repository whose R2 objects are still being deleted. */
6873 const PURGE_KEY = "purge";
6974
@@ -233,6 +238,65 @@
233238 if (opts?.live !== false && event.type !== "audit") this.broadcast("repo", { kind: "event", type: event.type });
234239 }
235240
241+ // ---------------------------------------------------------------- purging unreachable objects
242+
243+ /** Bumped by every purge; a push that started before one refuses its ref updates. */
244+ private purgeEpoch = 0;
245+
246+ /**
247+ * Delete every object no ref reaches, so a secret force-pushed away can't be fetched by its SHA any
248+ * more (git keeps them until a gc, and so did Diffinery, forever). Also kept: the commits of open
249+ * pull requests, of the merge queue and of unfinished workflow runs. Runs with every other request
250+ * held, so nothing writes a ref meanwhile.
251+ */
252+ async purgeUnreachable(caller: Caller): Promise<{ objects: number; bytes: number }> {
253+ if (caller.kind !== "user" || caller.permissions || !atLeast(caller.role, "admin")) throw new HttpError(403, "only repository admins can purge unreachable objects");
254+ const total = this.store.count();
255+ if (total > MAX_PURGE_OBJECTS) throw new HttpError(422, `this repository has ${total} objects; purging works for up to ${MAX_PURGE_OBJECTS}. Ask a site administrator`);
256+ return this.ctx.blockConcurrencyWhile(async () => {
257+ const roots = [
258+ ...this.sql.exec<{ oid: string }>("SELECT oid FROM refs").toArray().map((r) => r.oid),
259+ ...this.sql.exec<{ oid: string }>("SELECT head_sha AS oid FROM pulls UNION SELECT head_sha FROM merge_queue UNION SELECT group_sha FROM merge_queue WHERE group_sha IS NOT NULL UNION SELECT parent_sha FROM merge_queue WHERE parent_sha IS NOT NULL").toArray().map((r) => r.oid),
260+ ...this.sql.exec<{ oid: string }>("SELECT sha AS oid FROM runs WHERE status != 'completed' UNION SELECT head_sha FROM runs WHERE status != 'completed' AND head_sha IS NOT NULL").toArray().map((r) => r.oid),
261+ ];
262+ const keep = new Set<string>();
263+ const stack = [...roots];
264+ while (stack.length) {
265+ const oid = stack.pop()!;
266+ if (keep.has(oid)) continue;
267+ const row = this.sql.exec<{ type: number }>("SELECT type FROM objects WHERE oid = ?", oid).toArray()[0];
268+ if (!row) continue;
269+ keep.add(oid);
270+ const type = CODE_TYPES[row.type];
271+ if (type === "blob") continue;
272+ const obj = await this.store.read(oid);
273+ if (!obj) continue;
274+ if (type === "commit") {
275+ const c = parseCommit(obj.content);
276+ stack.push(c.tree, ...c.parents);
277+ } else if (type === "tree") {
278+ // A gitlink (160000) names a commit in another repository.
279+ for (const e of parseTree(obj.content)) if (e.mode !== "160000") stack.push(e.oid);
280+ } else if (type === "tag") {
281+ const target = /^object ([0-9a-f]{40})/.exec(fromUtf8(obj.content))?.[1];
282+ if (target) stack.push(target);
283+ }
284+ }
285+ const doomed = this.sql.exec<{ oid: string; r2: number; size: number }>("SELECT oid, r2, size FROM objects").toArray().filter((r) => !keep.has(r.oid));
286+ for (let i = 0; i < doomed.length; i += 100) {
287+ const chunk = doomed.slice(i, i + 100).map((r) => r.oid);
288+ this.sql.exec(`DELETE FROM objects WHERE oid IN (${chunk.map(() => "?").join(",")})`, ...chunk);
289+ }
290+ this.store.forget();
291+ this.purgeEpoch++;
292+ const r2 = doomed.filter((r) => r.r2).map((r) => this.store.r2KeyOf(r.oid));
293+ for (let i = 0; i < r2.length; i += 1000) await this.env.BUCKET.delete(r2.slice(i, i + 1000));
294+ // Cached zipballs and tarballs are rebuilt from what's left on the next download.
295+ await deletePrefixes(this.env.BUCKET, [archivePrefix(this.meta.repoId)], 50);
296+ return { objects: doomed.length, bytes: doomed.reduce((n, r) => n + r.size, 0) };
297+ });
298+ }
299+
236300 // ---------------------------------------------------------------- refs
237301
238302 refs(prefix = "refs/"): RefEntry[] {
@@ -317,6 +381,9 @@
317381 return reportStatus(req, "push exceeds 100 MB; split it into smaller pushes", []);
318382 }
319383 const req = parseReceivePack(body);
384+ // A purge of unreachable objects that ran while this push was in flight may have deleted objects
385+ // the push counted on the repository having: refuse rather than point a ref at them.
386+ const epoch = this.purgeEpoch;
320387 // `git push -o diffinery-import` from a GitHub import's own token: no workflows, webhooks,
321388 // pull request syncs or notifications (see apps/diffinery/src/imports).
322389 const importMode = !!caller.importId && req.pushOptions.some((o) => o.trim() === "diffinery-import");
@@ -401,6 +468,10 @@
401468 results.push({ ref: c.ref, ok: false, reason: "atomic transaction failed" });
402469 continue;
403470 }
471+ if (epoch !== this.purgeEpoch) {
472+ results.push({ ref: c.ref, ok: false, reason: "the repository's unreachable objects were purged during this push; push again" });
473+ continue;
474+ }
404475 const applied = this.casRef(c.cmd.ref, c.cmd.oldOid, c.cmd.newOid);
405476 results.push(applied ? { ref: c.ref, ok: true } : { ref: c.ref, ok: false, reason: "fetch first (stale info)" });
406477 if (applied && importMode) {
modifiedapps/diffinery/src/repo/objects.ts+11−0View file
@@ -24,6 +24,17 @@
2424 return `repos/${this.repoId}/objects/${oid.slice(0, 2)}/${oid.slice(2)}`;
2525 }
2626
27+ /** Where an object kept in R2 lives, for deleting it. */
28+ r2KeyOf(oid: string): string {
29+ return this.r2Key(oid);
30+ }
31+
32+ /** Drop the in-memory cache (after objects were deleted). */
33+ forget(): void {
34+ this.cache.clear();
35+ this.cacheBytes = 0;
36+ }
37+
2738 private remember(oid: string, obj: GitObject) {
2839 if (obj.content.length > 4 * 1024 * 1024) return;
2940 this.cache.set(oid, obj);
modifiedapps/diffinery/src/web/repo-settings.tsx+19−0View file
@@ -299,6 +299,14 @@
299299 )}
300300 </Form>
301301 <hr />
302+ <Form action={`${base}/settings/purge`} t={t} class="row between mb" confirm="Permanently delete every commit and file that no branch, tag or open pull request reaches?">
303+ <div>
304+ <b>Purge unreachable objects</b>
305+ <div class="muted small">After you force-push a secret away: permanently delete the commits and files no branch, tag or open pull request reaches, so nobody can fetch them by their SHA. Rotate the secret too.</div>
306+ </div>
307+ <button class="btn danger">Purge</button>
308+ </Form>
309+ <hr />
302310 <Form action={`${base}/settings/delete`} t={t} match={full}>
303311 <b>Delete this repository</b>
304312 <p class="muted small">
@@ -763,6 +771,17 @@
763771 );
764772 });
765773
774+ post(web, "/:owner/:repo/settings/purge", async (c) => {
775+ const l = await load(c);
776+ requireAccount(c);
777+ return act(c, `${repoUrl(l.repo)}/settings`, async () => {
778+ requireRecentSignIn(c, `${repoUrl(l.repo)}/settings`);
779+ const r = await l.stub.purgeUnreachable(l.caller);
780+ audit(c, l, "repo.purge_unreachable", { objects: r.objects, bytes: r.bytes });
781+ flash(c, "notice", r.objects ? `Purged ${r.objects} unreachable object${r.objects === 1 ? "" : "s"}.` : "Nothing to purge: every object is reachable.");
782+ });
783+ });
784+
766785 post(web, "/:owner/:repo/settings/transfer", async (c, f) => {
767786 const l = await load(c, { whileArchived: true });
768787 const me = requireAccount(c);
modifiedapps/diffinery/test/e2e/security.test.mjs+40−0View file
@@ -1981,3 +1981,43 @@
19811981 await ok("PUT", "/repos/guild/sec-b/contents/again.txt", { message: "again", content: b64("again\n") });
19821982 assert.deepEqual(await got("sec-b", false), ["EVERYWHERE", "ONLY_A", "PRIVATE_ONLY"]);
19831983 });
1984+
1985+test("security: an admin can purge objects a force-push left unreachable", async () => {
1986+ const R = `/repos/${OWNER}/purge-me`;
1987+ const mainSha = await repoWithReadme("purge-me");
1988+ await status(204, "PUT", `${R}/collaborators/rita`, { permission: "write" });
1989+ const clone = join(work, "purge-me");
1990+ const authed = `${base.replace("://", `://x:${T.olive}@`)}/${OWNER}/purge-me.git`;
1991+ execFileSync("git", ["clone", "-q", authed, clone], { stdio: "pipe" });
1992+ const g = (args) => execFileSync("git", ["-c", "user.name=Olive", "-c", "user.email=olive@example.com", ...args], { cwd: clone, stdio: "pipe" }).toString().trim();
1993+ // A secret pushed by mistake, then force-pushed away.
1994+ writeFileSync(join(clone, "leak.txt"), "AKIA-not-really-a-key-1234567890\n");
1995+ g(["add", "leak.txt"]);
1996+ g(["commit", "-q", "-m", "oops"]);
1997+ const leak = g(["rev-parse", "HEAD"]);
1998+ const blob = g(["rev-parse", "HEAD:leak.txt"]);
1999+ g(["push", "-q", "origin", "HEAD:main"]);
2000+ g(["push", "-q", "-f", "origin", `${mainSha}:main`]);
2001+ // Unreachable, but still served by its SHA.
2002+ await ok("GET", `${R}/git/commits/${leak}`);
2003+ await ok("GET", `${R}/git/blobs/${blob}`);
2004+
2005+ await status(403, "POST", `${R}/diffinery/purge-unreachable`, undefined, T.rita);
2006+ const purged = await ok("POST", `${R}/diffinery/purge-unreachable`);
2007+ assert.equal(purged.objects, 3, JSON.stringify(purged)); // the commit, its tree and the blob
2008+ await status(404, "GET", `${R}/git/commits/${leak}`);
2009+ await status(404, "GET", `${R}/git/blobs/${blob}`);
2010+ // Everything a branch reaches is still there, and a fresh clone checks out.
2011+ assert.equal((await ok("GET", `${R}/branches/main`)).commit.sha, mainSha);
2012+ const fresh = join(work, "purge-me-fresh");
2013+ execFileSync("git", ["clone", "-q", authed, fresh], { stdio: "pipe" });
2014+ execFileSync("git", ["fsck", "--full"], { cwd: fresh, stdio: "pipe" });
2015+ // A second purge finds nothing, and pushing works as before.
2016+ assert.equal((await ok("POST", `${R}/diffinery/purge-unreachable`)).objects, 0);
2017+ g(["reset", "-q", "--hard", mainSha]);
2018+ writeFileSync(join(clone, "ok.txt"), "fine\n");
2019+ g(["add", "ok.txt"]);
2020+ g(["commit", "-q", "-m", "fine"]);
2021+ g(["push", "-q", "origin", "HEAD:main"]);
2022+ assert.equal((await ok("GET", `${R}/branches/main`)).commit.sha, g(["rev-parse", "HEAD"]));
2023+});