Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Per-account storage quota (S11): count what repositories and packages store, refuse writes past it (#139)

Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo

Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent 2c79ea1commit 2eb67ab3e37827ed175487f4e9075bd6624e7558Browse files
24 files changed+464−50
Files (24)
modifiedapps/diffinery/docs/admin/site-administration.md+6−4View file
@@ -195,20 +195,22 @@
195195
196196 ## Account quotas
197197
198-Every person and organization has [quotas](/docs/reference/limits#account-quotas) on hosted minutes per month, hosted jobs running at once, agent sandboxes open at once and repositories. Accounts run by a site administrator have none. The defaults come from the `QUOTA_*` variables in [Configuration](/docs/self-hosting/configuration). There is no page for quotas; use the API:
198+Every person and organization has [quotas](/docs/reference/limits#account-quotas) on hosted minutes per month, hosted jobs running at once, agent sandboxes open at once, repositories and storage. Accounts run by a site administrator have none. The defaults come from the `QUOTA_*` variables in [Configuration](/docs/self-hosting/configuration). There is no page for quotas; use the API:
199199
200200 ```sh
201201 diffinery api GET /admin/accounts/acme/quotas
202-echo '{"hosted_minutes": 6000, "repositories": -1}' | diffinery api PATCH /admin/accounts/acme/quotas --input -
202+echo '{"hosted_minutes": 6000, "repositories": -1, "storage_bytes": 53687091200}' | diffinery api PATCH /admin/accounts/acme/quotas --input -
203203 ```
204204
205205 | Request | What it does | Who |
206206 | --- | --- | --- |
207-| `GET /api/v3/admin/accounts/{login}/quotas` | The account's quotas and what it uses: `hosted_minutes` (`used`, `limit`), `hosted_jobs` (`running`, `limit`), `sandboxes` (`open`, `limit`), `repositories` (`count`, `limit`), with `exempt` and `period_start` (the start of the month, UTC) | site administrator, any token scope |
208-| `PATCH /api/v3/admin/accounts/{login}/quotas` | Changes only the quotas you send: a whole number, `-1` for unlimited, or `null` to go back to the default. Answers like `GET`. Audited as `account.quotas.set`. | site administrator, with an `admin` token not limited to repositories, or a browser session |
207+| `GET /api/v3/admin/accounts/{login}/quotas` | The account's quotas and what it uses: `hosted_minutes` (`used`, `limit`), `hosted_jobs` (`running`, `limit`), `sandboxes` (`open`, `limit`), `repositories` (`count`, `limit`), `storage_bytes` (`used`, `limit`, in bytes), with `exempt` and `period_start` (the start of the month, UTC) | site administrator, any token scope |
208+| `PATCH /api/v3/admin/accounts/{login}/quotas` | Changes only the quotas you send: a whole number (`storage_bytes` in bytes, up to 1 PiB), `-1` for unlimited, or `null` to go back to the default. Answers like `GET`. Audited as `account.quotas.set`. | site administrator, with an `admin` token not limited to repositories, or a browser session |
209209
210210 `{login}` is a person or an organization; an agent gets `404 no person or organization with that name`, because agents use their owner's quotas. Anyone else gets `403 only site administrators can see and change quotas`. People see their own usage with `GET /api/v3/user/usage`, and organization owners and admins see their organization's with `GET /api/v3/orgs/{org}/usage`.
211211
212+Storage is counted per repository as it changes: each repository's Durable Object keeps a running total of its git objects, Git LFS objects and release assets and reports it to D1 (`repo_storage`), where the owner's repositories and packages are added up. A repository that hasn't reported yet, such as one made before storage was counted, is measured the first time its account's storage is checked (up to 25 per check): its git objects are added up once from its own database, and its Git LFS objects are listed from R2 once. Purging unreachable objects counts the repository again.
213+
212214 ## Audit log
213215
214216 **Settings → Audit log** (`/settings/audit`) shows the 200 most recent security-relevant events: sign-ins, passkeys, tokens, invites, account status changes, repository and access changes, pushes, merges and push protection. Repository and organization settings changed through the REST API or the CLI are recorded too: repository settings, archiving, collaborators, branch protection, Actions secrets and variables (repository, environment and organization), webhooks, a repository's workflow token settings, and deleted Actions artifacts and caches. Imports from GitHub (`import.create`, `import.push`, `import.finish` and the rest) and linked GitHub accounts are recorded too. Settings changes made through the API have `via: api` in their **Detail**. Each row has **When**, **Actor** (and who an agent acted for), **Action**, **Target** and **Detail**. See [Audit log](/docs/security/audit-log) for every action and how to use it.
modifiedapps/diffinery/docs/reference/limits.md+1−0View file
@@ -180,6 +180,7 @@
180180 | Hosted jobs at once | 5 | Across all the account's repositories. Further jobs wait as queued and start, oldest first, as running ones finish (checked at least once a minute). A job that waits 6 hours fails with "The job waited 6 hours for one of this account's hosted jobs to finish, and was not started." |
181181 | Agent sandboxes open at once | 5 | Across all the account's repositories, including agent tasks', reviews' and terminals' sandboxes. A new one is refused with `429 this account already has 5 agent sandboxes open; close one or wait for one to finish`. |
182182 | Repositories | 500 | Creating, importing or transferring in a repository past it is refused with `403 <account> has reached its limit of 500 repositories; delete one or ask a site administrator to raise it`. |
183+| Storage | 10 GB (10,737,418,240 bytes) | What the account's repositories store (git objects as stored, compressed; Git LFS objects; release assets) and its container images' blobs and manifests (a blob several of its images use counts once). Actions artifacts and caches don't count: they have their own limits per repository (see [Caches and artifacts](#caches-and-artifacts)) and expire. A write that would take the account past it is refused with `<account> has reached its storage limit of 10737418240 bytes (git objects, Git LFS objects, release assets and packages); delete some or ask a site administrator to raise it`: a push that brings new objects (git shows `remote: Push refused: ...` and rejects every ref with `storage limit reached`), Git LFS uploads (each object gets error `507` in the batch response), release asset uploads, files and blobs written through the API, the web editor or MCP (`403`), container image uploads (`403 DENIED`), and starting a GitHub import (`403`); an import that reaches it on the way stops with the same message. Reads, merges, deletes and pushes that only delete or move refs keep working. A deleted branch's objects stay stored until you [purge unreachable objects](/docs/security/push-protection#removing-a-secret-that-was-pushed); deleting releases, assets, image versions or the repository frees their bytes. Repositories made before this quota existed are measured the first time their account's storage is checked. |
183184
184185 ## Egress
185186
modifiedapps/diffinery/docs/repositories/git-lfs.md+1−0View file
@@ -177,6 +177,7 @@
177177 | Hash algorithm | SHA-256 |
178178 | File locking | Not supported |
179179 | Push protection | LFS file contents are not scanned for secrets, only the pointer files in the Git push. |
180+| Storage | LFS objects count toward the repository owner's [storage quota](/docs/reference/limits#account-quotas). Past it, uploads are refused: each object in the batch response gets error `507` with the reason. |
180181
181182 ## Troubleshooting
182183
modifiedapps/diffinery/docs/self-hosting/configuration.md+1−1View file
@@ -31,7 +31,7 @@
3131 | `SIGNUP` | Not set | No | `open` lets anyone create an account at `/signup`. See [SIGNUP](#signup). Any other value, or none, keeps the instance invite-only. |
3232 | `TURNSTILE_SITE_KEY` | Not set | With `SIGNUP=open` | The site key of the Cloudflare Turnstile widget on the sign-up page. See [SIGNUP](#signup). |
3333 | `IMPORT_MAX_REPO_KB` | Not set | No | The largest GitHub repository a one-repository import accepts, in KB as GitHub reports a repository's `size`. Default `10485760` (10 GB). An empty value, `0`, a negative number or text that isn't a number counts as not set. A larger source is refused before the import starts with `422 the repository is about N GB, over the M GB import limit; ask an administrator to raise IMPORT_MAX_REPO_KB or split the repository`. See [GitHub imports](#github-imports). |
34-| `QUOTA_HOSTED_MINUTES`, `QUOTA_HOSTED_JOBS`, `QUOTA_SANDBOXES`, `QUOTA_REPOSITORIES` | Not set | No | The default per-account quotas: hosted minutes per month (default `3000`), hosted Actions jobs at once (`5`), agent sandboxes open at once (`5`) and repositories (`500`). `-1` is unlimited; text that isn't a whole number counts as not set. Site administrators change them for one account with `PATCH /api/v3/admin/accounts/{login}/quotas`, and accounts run by a site administrator have none. See [Limits](/docs/reference/limits#account-quotas). |
34+| `QUOTA_HOSTED_MINUTES`, `QUOTA_HOSTED_JOBS`, `QUOTA_SANDBOXES`, `QUOTA_REPOSITORIES`, `QUOTA_STORAGE_BYTES` | Not set | No | The default per-account quotas: hosted minutes per month (default `3000`), hosted Actions jobs at once (`5`), agent sandboxes open at once (`5`), repositories (`500`) and storage in bytes (`10737418240`, 10 GB). `-1` is unlimited; text that isn't a whole number counts as not set. Site administrators change them for one account with `PATCH /api/v3/admin/accounts/{login}/quotas`, and accounts run by a site administrator have none. See [Limits](/docs/reference/limits#account-quotas). |
3535 | `IMPORT_GHE_HOST` | Not set | No | The host name of a GitHub Enterprise Server (like `ghe.example.com`) that imports may download release assets from. Without it, imports download release assets only from github.com and its asset hosts, and assets of an Enterprise import are skipped with a warning. |
3636 | `GITHUB_OAUTH_CLIENT_ID` | Not set | No | The client ID of a GitHub OAuth app, for linking GitHub accounts with a GitHub sign-in instead of a gist. Needs `GITHUB_OAUTH_CLIENT_SECRET` too. See [GitHub imports](#github-imports). |
3737 | `IMPORT_GIT_AGENT` | Not set in production; `external` in the `dev` environment | Never in production | With `external`, an import's Git transfer doesn't start a container: it waits for an import agent started by hand (see [The dev environment](#the-dev-environment)). |
addedapps/diffinery/migrations/0022_storage_quota.sql+10−0View file
@@ -1,0 +1,10 @@
1+-- What each repository stores for its owner's storage quota (lib/quotas.ts): git objects, Git LFS objects
2+-- and release assets, in bytes, as the repository's Durable Object last reported it. No row: not measured yet.
3+CREATE TABLE IF NOT EXISTS repo_storage (
4+ repo_id TEXT PRIMARY KEY REFERENCES repos(id) ON DELETE CASCADE,
5+ bytes INTEGER NOT NULL,
6+ updated_at TEXT NOT NULL
7+);
8+
9+ALTER TABLE account_quotas ADD COLUMN storage_bytes INTEGER;
10+
modifiedapps/diffinery/src/docs/api-meta.ts+2−1View file
@@ -139,7 +139,7 @@
139139 "GET /users/{login}": { summary: "Get an account", description: "A user, organization or agent by login.", access: "account" },
140140 "GET /user/usage": {
141141 summary: "Get your quotas and usage",
142- description: "Hosted minutes used this month, hosted jobs running, sandboxes open and repositories owned, each with its `limit` (`-1` is unlimited), and `exempt` for accounts run by a site administrator. An agent gets its owner's. See [Limits](/docs/reference/limits#account-quotas).",
142+ description: "Hosted minutes used this month, hosted jobs running, sandboxes open, repositories owned and bytes stored, each with its `limit` (`-1` is unlimited), and `exempt` for accounts run by a site administrator. An agent gets its owner's. See [Limits](/docs/reference/limits#account-quotas).",
143143 access: "account",
144144 },
145145 "GET /orgs/{org}/usage": { summary: "Get an organization's quotas and usage", description: "The same as `GET /user/usage`, for the organization's owners and admins.", access: "org-admin" },
@@ -153,6 +153,7 @@
153153 hosted_jobs: ["integer", "Hosted Actions jobs running at once."],
154154 sandboxes: ["integer", "Agent sandboxes open at once."],
155155 repositories: ["integer", "Repositories the account owns."],
156+ storage_bytes: ["integer", "Bytes stored: git objects, Git LFS objects, release assets and packages, not Actions artifacts or caches."],
156157 },
157158 },
158159 "GET /admin/repos/{owner}/{repo}/actions-suspension": {
modifiedapps/diffinery/src/env.ts+1−0View file
@@ -53,6 +53,7 @@
5353 QUOTA_HOSTED_JOBS?: string;
5454 QUOTA_SANDBOXES?: string;
5555 QUOTA_REPOSITORIES?: string;
56+ QUOTA_STORAGE_BYTES?: string;
5657 /** "web": hosted jobs and sandboxes reach only HTTP, HTTPS and DNS (lib/egress.ts). Anything else, or unset, is open. */
5758 EGRESS?: string;
5859 /** Largest source repository an import will take, in KB (GitHub `size`); default 10 GB. */
modifiedapps/diffinery/src/http/api.ts+1−0View file
@@ -92,6 +92,7 @@
9292 hosted_jobs: { running: u.hosted_jobs, limit: q.hosted_jobs },
9393 sandboxes: { open: u.sandboxes, limit: q.sandboxes },
9494 repositories: { count: u.repositories, limit: q.repositories },
95+ storage_bytes: { used: u.storage_bytes, limit: q.storage_bytes },
9596 };
9697 };
9798 api.get("/user/usage", async (c) => {
modifiedapps/diffinery/src/http/files.ts+1−1View file
@@ -28,7 +28,7 @@
2828 caller: Caller,
2929 ) {
3030 if (data.length !== null && data.length > MAX_ASSET_BYTES) throw new HttpError(413, TOO_LARGE);
31- const { id, r2Key } = await stub.startReleaseAsset(releaseId, meta, caller);
31+ const { id, r2Key } = await stub.startReleaseAsset(releaseId, { ...meta, size: data.length ?? undefined }, caller);
3232 try {
3333 let size: number;
3434 if (data.length !== null) {
modifiedapps/diffinery/src/http/git.ts+12−5View file
@@ -149,10 +149,11 @@
149149
150150 git.post("/:owner/:repo/info/lfs/objects/batch", async (c) => {
151151 const req = (await c.req.json()) as { operation: "upload" | "download"; objects: Array<{ oid: string; size: number }> };
152- const { repo } = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), req.operation === "upload" ? "write" : "read", { readOnlyPost: req.operation !== "upload", follow: true });
152+ const { repo, stub } = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), req.operation === "upload" ? "write" : "read", { readOnlyPost: req.operation !== "upload", follow: true });
153153 const base = `${c.env.PUBLIC_URL}/${c.req.param("owner")}/${c.req.param("repo")}/info/lfs/objects`;
154154 const auth = c.req.header("authorization");
155155 const objects = [];
156+ const uploads: Array<{ oid: string; size: number }> = [];
156157 for (const o of req.objects.slice(0, 1000)) {
157158 if (!/^[0-9a-f]{64}$/.test(o.oid)) {
158159 objects.push({ oid: o.oid, size: o.size, error: { code: 422, message: "invalid oid" } });
@@ -162,10 +163,12 @@
162163 if (req.operation === "download") {
163164 if (!head) objects.push({ oid: o.oid, size: o.size, error: { code: 404, message: "object not found" } });
164165 else objects.push({ oid: o.oid, size: head.size, authenticated: true, actions: { download: { href: `${base}/${o.oid}`, header: auth ? { Authorization: auth } : {}, expires_in: 3600 } } });
165- } else {
166- objects.push(head ? { oid: o.oid, size: o.size, authenticated: true } : { oid: o.oid, size: o.size, authenticated: true, actions: { upload: { href: `${base}/${o.oid}`, header: auth ? { Authorization: auth } : {}, expires_in: 3600 } } });
167- }
166+ } else if (head) objects.push({ oid: o.oid, size: o.size, authenticated: true });
167+ else uploads.push(o);
168168 }
169+ // Past the owner's storage quota (lib/quotas.ts) no new object is taken: each gets the reason instead.
170+ const full = uploads.length ? await stub.storageRefusal(uploads.reduce((n, o) => n + (Number(o.size) || 0), 0)) : null;
171+ for (const o of uploads) objects.push(full ? { oid: o.oid, size: o.size, error: { code: 507, message: full } } : { oid: o.oid, size: o.size, authenticated: true, actions: { upload: { href: `${base}/${o.oid}`, header: auth ? { Authorization: auth } : {}, expires_in: 3600 } } });
169172 return c.body(JSON.stringify({ transfer: "basic", objects, hash_algo: "sha256" }), 200, { "content-type": LFS_TYPE });
170173 });
171174
@@ -177,13 +180,17 @@
177180 });
178181
179182 git.put("/:owner/:repo/info/lfs/objects/:oid", async (c) => {
180- const { repo } = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), "write", { follow: true });
183+ const { repo, stub } = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), "write", { follow: true });
181184 const oid = c.req.param("oid");
182185 if (!/^[0-9a-f]{64}$/.test(oid)) throw new HttpError(422, "invalid oid");
186+ const length = Number(c.req.header("content-length"));
187+ const full = await stub.storageRefusal(Number.isFinite(length) ? length : 0);
188+ if (full) throw new HttpError(507, full);
183189 // Verify the content hash while storing: R2 checks the SHA-256 we pass.
184190 const body = await c.req.arrayBuffer();
185191 const digest = [...new Uint8Array(await crypto.subtle.digest("SHA-256", body))].map((b) => b.toString(16).padStart(2, "0")).join("");
186192 if (digest !== oid) throw new HttpError(422, "content does not match oid");
187193 await c.env.BUCKET.put(lfsKey(repo.id, oid), body);
194+ await stub.recordLfs(oid, body.byteLength);
188195 return c.body(null, 200);
189196 });
modifiedapps/diffinery/src/imports/service.ts+4−0View file
@@ -6,6 +6,7 @@
66 import { Directory, type Account, type Repo, type TokenScope } from "../lib/directory.js";
77 import { atLeast } from "../repo/types.js";
88 import { HttpError, newId, now } from "../lib/util.js";
9+import { checkStorageQuota } from "../lib/quotas.js";
910 import { GitHubClient, GitHubError, RateLimited, listPath } from "./github.js";
1011 import {
1112 People,
@@ -407,6 +408,9 @@
407408 if (actor.kind === "agent") req.options.secrets = "stop";
408409 // M5: cap how many imports one account or organization can run at once.
409410 await this.assertImportQuota(actor.id, owner.id, req.mode);
411+ // An import stores what it copies for the owner: refused up front past the owner's storage quota
412+ // (lib/quotas.ts); one that reaches it on the way fails with the quota's message.
413+ await checkStorageQuota(this.env, owner.id, owner.login);
410414 const gh = new GitHubClient(req.apiUrl, req.token);
411415 const stored: StoredRequest = {
412416 visibility: req.visibility,
modifiedapps/diffinery/src/lib/quotas.ts+57−9View file
@@ -9,18 +9,23 @@
99 * - hosted_jobs: hosted Actions jobs running at once across the account's repositories; more wait.
1010 * - sandboxes: agent sandboxes open at once across the account's repositories.
1111 * - repositories: repositories the account owns.
12+ * - storage_bytes: bytes the account stores: its repositories' git objects, Git LFS objects and release
13+ * assets, and its packages' blobs and manifests. Actions artifacts and caches don't count: each
14+ * repository has its own caps on them, and they expire.
1215 *
1316 * Defaults come from the environment (QUOTA_*) and otherwise DEFAULT_QUOTAS; a site administrator
1417 * can override them per account (account_quotas). -1 means unlimited. Accounts run by a site
1518 * administrator (one, or an organization with one as owner) have no quotas, so the instance's own
1619 * work never stops on them.
1720 */
18-export type QuotaKey = "hosted_minutes" | "hosted_jobs" | "sandboxes" | "repositories";
19-export const QUOTA_KEYS: readonly QuotaKey[] = ["hosted_minutes", "hosted_jobs", "sandboxes", "repositories"];
21+export type QuotaKey = "hosted_minutes" | "hosted_jobs" | "sandboxes" | "repositories" | "storage_bytes";
22+export const QUOTA_KEYS: readonly QuotaKey[] = ["hosted_minutes", "hosted_jobs", "sandboxes", "repositories", "storage_bytes"];
2023 export type Quotas = Record<QuotaKey, number>;
2124
22-export const DEFAULT_QUOTAS: Quotas = { hosted_minutes: 3000, hosted_jobs: 5, sandboxes: 5, repositories: 500 };
23-const ENV_KEYS: Record<QuotaKey, string> = { hosted_minutes: "QUOTA_HOSTED_MINUTES", hosted_jobs: "QUOTA_HOSTED_JOBS", sandboxes: "QUOTA_SANDBOXES", repositories: "QUOTA_REPOSITORIES" };
25+export const DEFAULT_QUOTAS: Quotas = { hosted_minutes: 3000, hosted_jobs: 5, sandboxes: 5, repositories: 500, storage_bytes: 10 * 1024 ** 3 };
26+const ENV_KEYS: Record<QuotaKey, string> = { hosted_minutes: "QUOTA_HOSTED_MINUTES", hosted_jobs: "QUOTA_HOSTED_JOBS", sandboxes: "QUOTA_SANDBOXES", repositories: "QUOTA_REPOSITORIES", storage_bytes: "QUOTA_STORAGE_BYTES" };
27+/** The largest quota a site administrator can set: storage is in bytes, the rest are counts. */
28+const MAX_QUOTA: Record<QuotaKey, number> = { hosted_minutes: 1e9, hosted_jobs: 1e9, sandboxes: 1e9, repositories: 1e9, storage_bytes: 2 ** 50 };
2429
2530 /**
2631 * The longest a job or sandbox can run (a job's 6 hours plus start-up, a sandbox's 24-hour token):
@@ -28,7 +33,7 @@
2833 */
2934 export const MAX_RUN_MS: Record<"job" | "sandbox", number> = { job: 7 * 3600_000, sandbox: 24 * 3600_000 };
3035
31-const UNLIMITED: Quotas = { hosted_minutes: -1, hosted_jobs: -1, sandboxes: -1, repositories: -1 };
36+const UNLIMITED: Quotas = { hosted_minutes: -1, hosted_jobs: -1, sandboxes: -1, repositories: -1, storage_bytes: -1 };
3237
3338 function envDefault(env: Env, k: QuotaKey): number {
3439 const raw = (env as unknown as Record<string, string | undefined>)[ENV_KEYS[k]];
@@ -66,6 +71,7 @@
6671 hosted_jobs: number;
6772 sandboxes: number;
6873 repositories: number;
74+ storage_bytes: number;
6975 }
7076
7177 /** What an account uses now. */
@@ -91,9 +97,51 @@
9197 }
9298 }
9399 const repos = await env.DB.prepare("SELECT count(*) AS n FROM repos WHERE owner_id = ?").bind(accountId).first<{ n: number }>();
94- return { hosted_minutes: minutes, hosted_jobs: jobs, sandboxes, repositories: repos?.n ?? 0 };
100+ return { hosted_minutes: minutes, hosted_jobs: jobs, sandboxes, repositories: repos?.n ?? 0, storage_bytes: await storedBytes(env, accountId) };
95101 }
96102
103+/** Repositories not measured yet (made before storage was counted) measured by one check. */
104+const MEASURE_AT_ONCE = 25;
105+
106+/**
107+ * Bytes the account stores: what its repositories last reported (repo_storage) and its packages' blobs
108+ * and manifests (a blob several of its packages use counts once). `live` is a repository counting its
109+ * own bytes, which are fresher than its report. Repositories that never reported are asked first, a
110+ * few per call, so storage made before it was counted is found without listing R2.
111+ */
112+export async function storedBytes(env: Env, accountId: string, live?: { repoId: string; bytes: number }): Promise<number> {
113+ const unmeasured = await env.DB.prepare("SELECT r.id FROM repos r LEFT JOIN repo_storage s ON s.repo_id = r.id WHERE r.owner_id = ? AND s.repo_id IS NULL AND r.id != ? LIMIT ?")
114+ .bind(accountId, live?.repoId ?? "", MEASURE_AT_ONCE)
115+ .all<{ id: string }>();
116+ await Promise.all(unmeasured.results.map((r) => env.REPO.get(env.REPO.idFromName(r.id)).reportStorage().catch((e: unknown) => console.error("could not measure a repository's storage", r.id, e))));
117+ const row = await env.DB.prepare(
118+ `SELECT (SELECT coalesce(sum(s.bytes), 0) FROM repo_storage s JOIN repos r ON r.id = s.repo_id WHERE r.owner_id = ?1 AND r.id != ?2)
119+ + (SELECT coalesce(sum(b.size), 0) FROM registry_blobs b WHERE b.digest IN (SELECT pb.digest FROM package_blobs pb JOIN packages p ON p.id = pb.package_id WHERE p.owner_id = ?1))
120+ + (SELECT coalesce(sum(m.size), 0) FROM package_manifests m JOIN packages p ON p.id = m.package_id WHERE p.owner_id = ?1) AS n`,
121+ )
122+ .bind(accountId, live?.repoId ?? "")
123+ .first<{ n: number }>();
124+ return (row?.n ?? 0) + (live?.bytes ?? 0);
125+}
126+
127+/**
128+ * Why a write of `adding` more bytes (0 when not known in advance) must be refused, or null: it would
129+ * take the account past its storage quota. Reads and deletes never ask, so an account past its quota
130+ * can always get back under it.
131+ */
132+export async function storageRefusal(env: Env, accountId: string, login: string, adding = 0, live?: { repoId: string; bytes: number }): Promise<string | null> {
133+ const q = await quotasFor(env, accountId);
134+ if (q.exempt || q.storage_bytes < 0) return null;
135+ if ((await storedBytes(env, accountId, live)) + adding <= q.storage_bytes) return null;
136+ return `${login} has reached its storage limit of ${q.storage_bytes} bytes (git objects, Git LFS objects, release assets and packages); delete some or ask a site administrator to raise it`;
137+}
138+
139+/** Refuse a write past the account's storage quota (see storageRefusal). */
140+export async function checkStorageQuota(env: Env, accountId: string, login: string, adding = 0): Promise<void> {
141+ const refused = await storageRefusal(env, accountId, login, adding);
142+ if (refused) throw new HttpError(403, refused);
143+}
144+
97145 const within = (limit: number, used: number) => limit < 0 || used < limit;
98146
99147 /**
@@ -133,13 +181,13 @@
133181 export async function setQuotas(env: Env, accountId: string, patch: Partial<Record<QuotaKey, number | null>>, by: string): Promise<void> {
134182 for (const [k, v] of Object.entries(patch)) {
135183 if (!QUOTA_KEYS.includes(k as QuotaKey)) throw new HttpError(422, `unknown quota ${k}; use ${QUOTA_KEYS.join(", ")}`);
136- if (v !== null && (typeof v !== "number" || !Number.isInteger(v) || v < -1 || v > 1e9)) throw new HttpError(422, `${k} must be a whole number, -1 for unlimited, or null for the default`);
184+ if (v !== null && (typeof v !== "number" || !Number.isInteger(v) || v < -1 || v > MAX_QUOTA[k as QuotaKey])) throw new HttpError(422, `${k} must be a whole number, -1 for unlimited, or null for the default`);
137185 }
138186 const cur = (await env.DB.prepare("SELECT * FROM account_quotas WHERE account_id = ?").bind(accountId).first<Record<QuotaKey, number | null>>()) ?? null;
139187 const next = Object.fromEntries(QUOTA_KEYS.map((k) => [k, k in patch ? (patch[k] ?? null) : (cur?.[k] ?? null)])) as Record<QuotaKey, number | null>;
140188 await env.DB.prepare(
141- "INSERT INTO account_quotas (account_id, hosted_minutes, hosted_jobs, sandboxes, repositories, updated_at, updated_by) VALUES (?, ?, ?, ?, ?, ?, ?) ON CONFLICT(account_id) DO UPDATE SET hosted_minutes = excluded.hosted_minutes, hosted_jobs = excluded.hosted_jobs, sandboxes = excluded.sandboxes, repositories = excluded.repositories, updated_at = excluded.updated_at, updated_by = excluded.updated_by",
189+ "INSERT INTO account_quotas (account_id, hosted_minutes, hosted_jobs, sandboxes, repositories, storage_bytes, updated_at, updated_by) VALUES (?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(account_id) DO UPDATE SET hosted_minutes = excluded.hosted_minutes, hosted_jobs = excluded.hosted_jobs, sandboxes = excluded.sandboxes, repositories = excluded.repositories, storage_bytes = excluded.storage_bytes, updated_at = excluded.updated_at, updated_by = excluded.updated_by",
142190 )
143- .bind(accountId, next.hosted_minutes, next.hosted_jobs, next.sandboxes, next.repositories, now(), by)
191+ .bind(accountId, next.hosted_minutes, next.hosted_jobs, next.sandboxes, next.repositories, next.storage_bytes, now(), by)
144192 .run();
145193 }
modifiedapps/diffinery/src/registry/http.ts+9−0View file
@@ -3,6 +3,7 @@
33 import type { AppContext, AppVars } from "../auth/principal.js";
44 import type { Account } from "../lib/directory.js";
55 import { HttpError } from "../lib/util.js";
6+import { checkStorageQuota } from "../lib/quotas.js";
67 import { allows, packageAccess, type PackageAccess, type PackageRole } from "./access.js";
78 import { DIGEST_RE, NAME_RE, RegistryStore, TAG_RE, manifestKey, type ManifestRow, type PackageRow, type UploadRow } from "./store.js";
89 import { UploadError, digestOf, discard, ingest } from "./upload.js";
@@ -225,8 +226,14 @@
225226 return { location: uploadUrl(t, u.id), range: `0-${Math.max(0, u.size - 1)}`, "docker-upload-uuid": u.id, "diffinery-upload-size": String(u.size), "content-length": "0" };
226227 }
227228
229+/** Refuse a blob past the owner's storage quota (lib/quotas.ts): `sofar` bytes uploaded, plus this request's. */
230+function checkStorage(c: AppContext, t: Target, sofar = 0): Promise<void> {
231+ return checkStorageQuota(c.env, t.owner.id, t.owner.login, sofar + (Number(c.req.header("content-length")) || 0));
232+}
233+
228234 async function startUpload(c: AppContext, full: string): Promise<Response> {
229235 const t = await target(c, full, "push");
236+ await checkStorage(c, t);
230237 const pkg = await ensurePackage(c, t);
231238 const mount = c.req.query("mount");
232239 const from = c.req.query("from");
@@ -278,6 +285,7 @@
278285 async function patchUpload(c: AppContext, full: string, id: string): Promise<Response> {
279286 const { t, u } = await loadUpload(c, full, id);
280287 checkContentRange(c, t, u);
288+ await checkStorage(c, t, u.size);
281289 const next = await ingest(c.env, u, c.req.raw.body, false);
282290 await t.store.saveUpload(next);
283291 return new Response(null, { status: 202, headers: uploadHeaders(t, next) });
@@ -288,6 +296,7 @@
288296 const digest = c.req.query("digest");
289297 if (!digest || !DIGEST_RE.test(digest)) throw new RegistryError(400, "DIGEST_INVALID", "the digest query parameter must be a sha256 digest");
290298 if (c.req.raw.body) checkContentRange(c, t, u);
299+ await checkStorage(c, t, u.size);
291300 return finishUpload(c, t, pkg, await ingest(c.env, u, c.req.raw.body, true), digest);
292301 }
293302
modifiedapps/diffinery/src/repo/RepoDO.ts+80−4View file
@@ -48,6 +48,7 @@
4848 import { scanForSecrets, secretBypassReason, type SecretHit } from "../git/secret-scan.js";
4949 import { HttpError, isValidRefName, now } from "../lib/util.js";
5050 import { archivePrefix, deletePrefixes, repoPrefixes, type SpilledBody } from "../lib/storage-keys.js";
51+import { storageRefusal } from "../lib/quotas.js";
5152 import { Actions } from "./actions.js";
5253 import { AutoMerge, isPerson } from "./auto-merge.js";
5354 import { Collab } from "./collab.js";
@@ -123,7 +124,7 @@
123124 this.sql.exec(REPO_SCHEMA);
124125 migrateRepoSchema(this.sql);
125126 const meta = this.readMeta();
126- if (meta) this.store = new RepoObjectStore(this.sql, env.BUCKET, meta.repoId);
127+ if (meta) this.store = new RepoObjectStore(this.sql, env.BUCKET, meta.repoId, () => this.storageChanged());
127128 this.collab = new Collab(this);
128129 this.actions = new Actions(this, env);
129130 this.queue = new MergeQueue(this);
@@ -151,7 +152,7 @@
151152 if (!this.readMeta()) this.policy.initNewRepo();
152153 this.sql.exec("INSERT OR REPLACE INTO meta (key, value) VALUES ('meta', ?)", JSON.stringify(meta));
153154 this.metaCache = meta;
154- this.store = new RepoObjectStore(this.sql, this.env.BUCKET, meta.repoId);
155+ this.store = new RepoObjectStore(this.sql, this.env.BUCKET, meta.repoId, () => this.storageChanged());
155156 }
156157
157158 async updateMeta(patch: Partial<RepoMeta>): Promise<RepoMeta> {
@@ -304,6 +305,8 @@
304305 this.sql.exec(`DELETE FROM objects WHERE oid IN (${chunk.map(() => "?").join(",")})`, ...chunk);
305306 }
306307 this.store.forget();
308+ this.store.recount();
309+ this.storageChanged();
307310 this.purgeEpoch++;
308311 const r2 = doomed.filter((r) => r.r2).map((r) => this.store.r2KeyOf(r.oid));
309312 for (let i = 0; i < r2.length; i += 1000) await this.env.BUCKET.delete(r2.slice(i, i + 1000));
@@ -313,6 +316,70 @@
313316 });
314317 }
315318
319+ // ---------------------------------------------------------------- storage (the owner's storage quota, lib/quotas.ts)
320+
321+ /** The report to D1 waiting to run, and the last one started: reports run one at a time. */
322+ private storageNext: Promise<void> | null = null;
323+ private storageLast: Promise<void> = Promise.resolve();
324+
325+ /** Bytes the repository stores for its owner: git objects, Git LFS objects and release assets. */
326+ async storageBytes(): Promise<number> {
327+ await this.listLfs();
328+ const q = (s: string) => this.sql.exec<{ n: number }>(s).one().n;
329+ return this.store.sizeBytes() + q("SELECT coalesce(sum(size), 0) AS n FROM lfs_objects") + q("SELECT coalesce(sum(size), 0) AS n FROM release_assets");
330+ }
331+
332+ /**
333+ * Tell D1 what the repository stores now (repo_storage), where the owner's quota adds it up. Calls
334+ * made while a report waits to run share it; one made while a report is under way gets the next.
335+ */
336+ reportStorage(): Promise<void> {
337+ this.storageNext ??= this.storageLast.then(async () => {
338+ this.storageNext = null;
339+ const bytes = await this.storageBytes();
340+ await this.env.DB.prepare("INSERT INTO repo_storage (repo_id, bytes, updated_at) VALUES (?, ?, ?) ON CONFLICT(repo_id) DO UPDATE SET bytes = excluded.bytes, updated_at = excluded.updated_at")
341+ .bind(this.meta.repoId, bytes, now())
342+ .run();
343+ });
344+ const next = this.storageNext;
345+ this.storageLast = next.catch(() => {});
346+ return next;
347+ }
348+
349+ /** What the repository stores changed: report it in the background. */
350+ storageChanged(): void {
351+ this.background(this.reportStorage());
352+ }
353+
354+ /** Why storing `adding` more bytes here must be refused (past the owner's storage quota), or null. */
355+ async storageRefusal(adding = 0): Promise<string | null> {
356+ const m = this.meta;
357+ return storageRefusal(this.env, m.ownerId, m.owner, adding, { repoId: m.repoId, bytes: await this.storageBytes() });
358+ }
359+
360+ /** Refuse storing `adding` more bytes past the owner's storage quota. */
361+ async checkStorage(adding = 0): Promise<void> {
362+ const refused = await this.storageRefusal(adding);
363+ if (refused) throw new HttpError(403, refused);
364+ }
365+
366+ /** A Git LFS object was stored in R2 (http/git.ts). */
367+ recordLfs(oid: string, size: number): void {
368+ if (this.sql.exec("INSERT OR IGNORE INTO lfs_objects (oid, size) VALUES (?, ?) RETURNING oid", oid, size).toArray().length) this.storageChanged();
369+ }
370+
371+ /** Git LFS objects stored before lfs_objects existed: listed from R2, once. */
372+ private async listLfs(): Promise<void> {
373+ if (this.sql.exec("SELECT 1 FROM counters WHERE name = 'lfs_listed'").toArray().length) return;
374+ let cursor: string | undefined;
375+ do {
376+ const page = await this.env.BUCKET.list({ prefix: `repos/${this.meta.repoId}/lfs/`, cursor, limit: 1000 });
377+ for (const o of page.objects) this.sql.exec("INSERT OR IGNORE INTO lfs_objects (oid, size) VALUES (?, ?)", o.key.slice(o.key.lastIndexOf("/") + 1), o.size);
378+ cursor = page.truncated ? page.cursor : undefined;
379+ } while (cursor);
380+ this.sql.exec("INSERT OR IGNORE INTO counters (name, value) VALUES ('lfs_listed', 1)");
381+ }
382+
316383 // ---------------------------------------------------------------- refs
317384
318385 refs(prefix = "refs/"): RefEntry[] {
@@ -488,6 +555,13 @@
488555 }
489556 }
490557
558+ // Past the owner's storage quota nothing new is stored; a push that only deletes or moves refs still works.
559+ const full = objects.some((o) => !this.store.has(o.oid)) ? await this.storageRefusal(req.pack?.length ?? 0) : null;
560+ if (full) {
561+ for (const c of req.commands) results.push({ ref: c.ref, ok: false, reason: "storage limit reached" });
562+ return reportStatus(req, null, results, [...messages, `Push refused: ${full}`]);
563+ }
564+
491565 try {
492566 for (const o of objects) {
493567 if (o.deflated) await this.store.writeDeflated(o.oid, o.type, o.size!, o.deflated);
@@ -782,6 +856,8 @@
782856 const why = ch.delete ? null : unsafeTreePath(ch.path);
783857 if (why) throw new HttpError(422, `invalid path ${JSON.stringify(ch.path)}: ${why}`);
784858 }
859+ const adding = input.changes.reduce((n, ch) => n + (ch.content?.length ?? 0), 0);
860+ if (adding) await this.checkStorage(adding);
785861 this.protectCommit(input.changes, input.allowSecrets, caller, input.message);
786862 const baseTree = parent ? (await readCommit(this.store, parent)).tree : null;
787863 const tree = await applyChanges(this.store, baseTree, input.changes);
@@ -1498,8 +1574,8 @@
14981574 refsWithTypes(...a: Parameters<GitData["refsWithTypes"]>) {
14991575 return this.gitdata.refsWithTypes(...a);
15001576 }
1501- createBlob(...a: Parameters<GitData["createBlob"]>) {
1502- return this.gitdata.createBlob(...a);
1577+ createBlob(content: Uint8Array, caller: Caller) {
1578+ return this.gitdata.createBlob(content, caller);
15031579 }
15041580 getBlob(...a: Parameters<GitData["getBlob"]>) {
15051581 return this.gitdata.getBlob(...a);
modifiedapps/diffinery/src/repo/gitdata.ts+5−2View file
@@ -130,8 +130,9 @@
130130
131131 // ---------------------------------------------------------------- blobs
132132
133- async createBlob(content: Uint8Array, caller: Caller): Promise<{ sha: string; size: number }> {
133+ async createBlob(content: Uint8Array, caller: Caller, quotaChecked = false): Promise<{ sha: string; size: number }> {
134134 this.requireWrite(caller);
135+ if (!quotaChecked) await this.repo.checkStorage(content.length);
135136 const oid = await hashObject("blob", content);
136137 this.scan(oid, content, caller);
137138 await this.store.writeKnown(oid, "blob", content);
@@ -150,6 +151,8 @@
150151 async createTree(baseTree: string | undefined, entries: TreeInput[], caller: Caller): Promise<string> {
151152 this.requireWrite(caller);
152153 if (!Array.isArray(entries)) throw new HttpError(422, "tree must be an array", { errors: [{ resource: "Tree", code: "invalid", field: "tree" }] });
154+ const adding = entries.reduce((n, e) => n + (e?.content === undefined ? 0 : String(e.content).length), 0);
155+ if (adding) await this.repo.checkStorage(adding);
153156 if (baseTree && this.objectType(baseTree) !== "tree") throw new HttpError(422, "base_tree is not a valid tree oid", { errors: [{ resource: "Tree", code: "invalid", field: "base_tree" }] });
154157 type Node = { entries: Map<string, TreeEntry>; children: Map<string, Node> };
155158 const load = async (oid: string | null): Promise<Node> => ({ entries: new Map(oid ? (await readTree(this.store, oid)).map((e) => [e.name, e]) : []), children: new Map() });
@@ -183,7 +186,7 @@
183186 let oid: string;
184187 if (e.content !== undefined) {
185188 if (mode === "040000" || mode === "160000") throw new HttpError(422, "content can only be used with blob modes");
186- oid = (await this.createBlob(new TextEncoder().encode(String(e.content)), caller)).sha;
189+ oid = (await this.createBlob(new TextEncoder().encode(String(e.content)), caller, true)).sha;
187190 } else {
188191 oid = String(e.sha ?? "");
189192 if (!isOid(oid)) throw new HttpError(422, `invalid sha for ${e.path}`, { errors: [{ resource: "Tree", code: "invalid", field: "sha" }] });
modifiedapps/diffinery/src/repo/objects.ts+31−15View file
@@ -19,7 +19,8 @@
1919 private cache = new Map<string, GitObject>();
2020 private cacheBytes = 0;
2121
22- constructor(private sql: SqlStorage, private bucket: R2Bucket, private repoId: string) {}
22+ /** `onAdded` is told when objects were stored (for the owner's storage quota, lib/quotas.ts). */
23+ constructor(private sql: SqlStorage, private bucket: R2Bucket, private repoId: string, private onAdded?: () => void) {}
2324
2425 private r2Key(oid: string) {
2526 return `repos/${this.repoId}/objects/${oid.slice(0, 2)}/${oid.slice(2)}`;
@@ -94,12 +95,9 @@
9495 async writeKnown(oid: string, type: ObjectType, content: Uint8Array): Promise<void> {
9596 if (this.has(oid)) return;
9697 const deflated = deflate(content);
97- if (deflated.length > INLINE_LIMIT) {
98- await this.bucket.put(this.r2Key(oid), deflated);
99- this.sql.exec("INSERT OR IGNORE INTO objects (oid, type, size, data, r2) VALUES (?, ?, ?, NULL, 1)", oid, TYPE_CODES[type], content.length);
100- } else {
101- this.sql.exec("INSERT OR IGNORE INTO objects (oid, type, size, data, r2) VALUES (?, ?, ?, ?, 0)", oid, TYPE_CODES[type], content.length, deflated);
102- }
98+ const r2 = deflated.length > INLINE_LIMIT;
99+ if (r2) await this.bucket.put(this.r2Key(oid), deflated);
100+ this.insert(oid, type, content.length, deflated.length, r2 ? null : deflated);
103101 this.remember(oid, { type, content });
104102 }
105103
@@ -109,22 +107,40 @@
109107 */
110108 async writeDeflated(oid: string, type: ObjectType, size: number, deflated: Uint8Array): Promise<void> {
111109 if (this.has(oid)) return;
112- if (deflated.length > INLINE_LIMIT) {
113- await this.bucket.put(this.r2Key(oid), sized(slices(deflated), deflated.length));
114- this.sql.exec("INSERT OR IGNORE INTO objects (oid, type, size, data, r2) VALUES (?, ?, ?, NULL, 1)", oid, TYPE_CODES[type], size);
115- } else {
116- this.sql.exec("INSERT OR IGNORE INTO objects (oid, type, size, data, r2) VALUES (?, ?, ?, ?, 0)", oid, TYPE_CODES[type], size, deflated.slice());
117- }
110+ const r2 = deflated.length > INLINE_LIMIT;
111+ if (r2) await this.bucket.put(this.r2Key(oid), sized(slices(deflated), deflated.length));
112+ this.insert(oid, type, size, deflated.length, r2 ? null : deflated.slice());
118113 }
119114
115+ /** Record an object (`data` null: it is in R2) and count its bytes, unless a concurrent write recorded it first. */
116+ private insert(oid: string, type: ObjectType, size: number, stored: number, data: Uint8Array | null): void {
117+ const added = this.sql.exec("INSERT OR IGNORE INTO objects (oid, type, size, data, r2, stored) VALUES (?, ?, ?, ?, ?, ?) RETURNING oid", oid, TYPE_CODES[type], size, data, data ? 0 : 1, stored).toArray().length;
118+ if (!added) return;
119+ this.sql.exec("UPDATE counters SET value = value + ? WHERE name = 'stored_bytes'", stored);
120+ this.onAdded?.();
121+ }
122+
120123 count(): number {
121124 return this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM objects").one().n;
122125 }
123126
124- /** Approximate stored bytes (compressed). */
127+ /**
128+ * Bytes the objects take as stored (compressed), kept as a running total. It is added up once when
129+ * missing: in a repository from before it was kept (R2 objects stored then count at their inflated
130+ * size), and after a purge deleted objects.
131+ */
125132 sizeBytes(): number {
126- return this.sql.exec<{ n: number | null }>("SELECT sum(length(data)) AS n FROM objects").one().n ?? 0;
133+ const kept = this.sql.exec<{ value: number }>("SELECT value FROM counters WHERE name = 'stored_bytes'").toArray()[0];
134+ if (kept) return kept.value;
135+ const n = this.sql.exec<{ n: number }>("SELECT coalesce(sum(coalesce(stored, CASE WHEN r2 = 1 THEN size ELSE length(data) END)), 0) AS n FROM objects").one().n;
136+ this.sql.exec("INSERT INTO counters (name, value) VALUES ('stored_bytes', ?)", n);
137+ return n;
127138 }
139+
140+ /** Forget the running total (objects were deleted); the next sizeBytes adds them up again. */
141+ recount(): void {
142+ this.sql.exec("DELETE FROM counters WHERE name = 'stored_bytes'");
143+ }
128144 }
129145
130146 /** `data` as a stream of 1 MiB views. */
modifiedapps/diffinery/src/repo/releases.ts+11−4View file
@@ -385,6 +385,7 @@
385385 this.sql.exec("DELETE FROM releases WHERE id = ?", id);
386386 this.sql.exec("DELETE FROM reactions WHERE subject = ?", `release:${id}`);
387387 if (this.pinned() === id) this.pin(null);
388+ if (keys.length) this.repo.storageChanged();
388389 await this.deleteObjects(keys);
389390 await this.announce(rel, ["deleted"], caller);
390391 }
@@ -496,6 +497,7 @@
496497 this.requireWrite(caller);
497498 const a = this.assetRow(id, caller);
498499 this.sql.exec("DELETE FROM release_assets WHERE id = ?", id);
500+ this.repo.storageChanged();
499501 await this.deleteObjects([a.r2_key]);
500502 }
501503
@@ -510,10 +512,14 @@
510512 if (mine.length) await this.env.BUCKET.delete(mine).catch((e) => console.warn("release asset cleanup failed", e));
511513 }
512514
513- /** Reserve an asset row; the Worker streams the bytes to the returned R2 key, then calls finishAsset. */
514- async startAsset(releaseId: number, input: { name: string; label?: string; contentType?: string }, caller: Caller, imported?: { createdAt?: string; uploader?: { id: string; login: string; kind: string } }): Promise<{ id: number; r2Key: string }> {
515+ /**
516+ * Reserve an asset row; the Worker streams the bytes to the returned R2 key, then calls finishAsset.
517+ * `size` is the asset's, when known: past the owner's storage quota no asset is stored.
518+ */
519+ async startAsset(releaseId: number, input: { name: string; label?: string; contentType?: string; size?: number }, caller: Caller, imported?: { createdAt?: string; uploader?: { id: string; login: string; kind: string } }): Promise<{ id: number; r2Key: string }> {
515520 this.requireWrite(caller);
516521 this.row(releaseId);
522+ await this.repo.checkStorage(input.size ?? 0);
517523 const name = assetName(input.name ?? "");
518524 const clash = this.sql.exec<{ id: number; state: string; r2_key: string }>("SELECT id, state, r2_key FROM release_assets WHERE release_id = ? AND name = ?", releaseId, name).toArray()[0];
519525 if (clash?.state === "uploaded") throw new HttpError(422, `an asset named ${name} already exists`, { errors: [{ resource: "ReleaseAsset", code: "already_exists", field: "name" }] });
@@ -546,6 +552,7 @@
546552 const a = this.sql.exec<AssetRow>("SELECT * FROM release_assets WHERE id = ?", id).toArray()[0];
547553 if (!a) throw new HttpError(404, "asset not found");
548554 this.sql.exec("UPDATE release_assets SET state = 'uploaded', size = ?, download_count = ?, updated_at = ? WHERE id = ?", size, extra.downloadCount ?? 0, extra.updatedAt ?? now(), id);
555+ this.repo.storageChanged();
549556 return toAsset(this.sql.exec<AssetRow>("SELECT * FROM release_assets WHERE id = ?", id).one());
550557 }
551558
@@ -645,7 +652,7 @@
645652 await obj.body.cancel();
646653 throw new HttpError(413, "asset is too large to import");
647654 }
648- const { id, r2Key } = await this.startAsset(releaseId, { name: meta.name, label: meta.label, contentType: meta.content_type }, caller, { createdAt: meta.created_at, uploader });
655+ const { id, r2Key } = await this.startAsset(releaseId, { name: meta.name, label: meta.label, contentType: meta.content_type, size: obj.size }, caller, { createdAt: meta.created_at, uploader });
649656 try {
650657 const fixed = new FixedLengthStream(obj.size);
651658 await Promise.all([obj.body.pipeTo(fixed.writable), this.env.BUCKET.put(r2Key, fixed.readable, { httpMetadata: obj.httpMetadata })]);
@@ -699,7 +706,7 @@
699706 await body.cancel();
700707 throw new HttpError(413, "asset is too large to import");
701708 }
702- const { id, r2Key } = await this.startAsset(releaseId, { name: meta.name, label: meta.label, contentType: meta.content_type ?? res.headers.get("content-type") ?? undefined }, caller, { createdAt: meta.created_at, uploader });
709+ const { id, r2Key } = await this.startAsset(releaseId, { name: meta.name, label: meta.label, contentType: meta.content_type ?? res.headers.get("content-type") ?? undefined, size: Number.isFinite(length) ? length : undefined }, caller, { createdAt: meta.created_at, uploader });
703710 try {
704711 let size: number;
705712 if (Number.isFinite(length) && length >= 0) {
modifiedapps/diffinery/src/repo/schema.ts+4−0View file
@@ -6,6 +6,8 @@
66 oid TEXT PRIMARY KEY, type INTEGER NOT NULL, size INTEGER NOT NULL, data BLOB, r2 INTEGER NOT NULL DEFAULT 0
77 );
88 CREATE TABLE IF NOT EXISTS counters (name TEXT PRIMARY KEY, value INTEGER NOT NULL);
9+-- Git LFS objects stored in R2 (repos/<id>/lfs/...), for the owner's storage quota (lib/quotas.ts).
10+CREATE TABLE IF NOT EXISTS lfs_objects (oid TEXT PRIMARY KEY, size INTEGER NOT NULL);
911
1012 CREATE TABLE IF NOT EXISTS issues (
1113 number INTEGER PRIMARY KEY, kind TEXT NOT NULL, title TEXT NOT NULL, body TEXT NOT NULL DEFAULT '',
@@ -222,6 +224,8 @@
222224 ["jobs", "gate_state", "TEXT"],
223225 // Logins that started earlier attempts of a re-run (JSON list): prevent_self_review covers them too.
224226 ["runs", "prior_actors", "TEXT"],
227+ // Bytes an object takes as stored (compressed, in SQLite or R2); null in objects stored before it was kept.
228+ ["objects", "stored", "INTEGER"],
225229 ];
226230
227231 export function migrateRepoSchema(sql: SqlStorage): void {
modifiedapps/diffinery/test/e2e/security.test.mjs+77−0View file
@@ -2408,6 +2408,83 @@
24082408 await until("the held job to leave the queue", async () => (await ok("GET", `${R}/actions/runs/${held.run}/jobs`, undefined, T.rita)).jobs[0].status !== "queued", 120_000);
24092409 });
24102410
2411+test("security: a per-account storage quota refuses pushes, Git LFS and release asset uploads past it; reads and deletes keep working", async () => {
2412+ const R = "/repos/milo/hoard";
2413+ await ok("POST", "/user/repos", { name: "hoard" }, T.milo);
2414+ const usage = async () => (await ok("GET", "/user/usage", undefined, T.milo)).storage_bytes;
2415+ assert.equal((await usage()).limit, 10 * 1024 ** 3);
2416+ // Push main and a topic branch, and upload a release asset, while there is room.
2417+ const clone = join(work, "hoard");
2418+ const remote = `${base.replace("://", "://x:" + T.milo + "@")}/milo/hoard.git`;
2419+ execFileSync("git", ["init", "-q", "-b", "main", clone]);
2420+ const commit = (name) => {
2421+ writeFileSync(join(clone, name), randomBytes(2000).toString("hex"));
2422+ gitRun(["add", name], clone);
2423+ gitRun(["commit", "-q", "-m", name], clone);
2424+ };
2425+ const push = (...args) => spawnSync("git", ["-c", "protocol.version=2", "push", remote, ...args], { cwd: clone, encoding: "utf8" });
2426+ commit("a.txt");
2427+ assert.equal(push("HEAD:main", "HEAD:refs/heads/topic").status, 0);
2428+ // Repositories report what they store as it changes: wait until the count stops moving.
2429+ const settled = async (what, test) => {
2430+ let last = -1;
2431+ return until(what, async () => {
2432+ const u = (await usage()).used;
2433+ if (u === last && test(u)) return u;
2434+ last = u;
2435+ await sleep(1000);
2436+ return null;
2437+ });
2438+ };
2439+ const pushed = await settled("the push to be counted", (u) => u > 2000);
2440+ const rel = await ok("POST", `${R}/releases`, { tag_name: "v1", target_commitish: "main" }, T.milo);
2441+ const upload = (bytes) => fetch(`${base}/api/uploads/repos/milo/hoard/releases/${rel.id}/assets?name=${bytes.length}.bin`, { method: "POST", headers: { authorization: `Bearer ${T.milo}`, "content-type": "application/octet-stream" }, body: bytes });
2442+ const asset = await (await upload(randomBytes(3000))).json();
2443+ assert.equal(asset.size, 3000);
2444+ const used = await settled("the asset to be counted", (u) => u === pushed + 3000);
2445+ assert.equal((await ok("GET", "/admin/accounts/milo/quotas", undefined, T.sully)).storage_bytes.used, used);
2446+
2447+ // ------------------------------------------------------------ at the limit: nothing new is stored
2448+ const set = await ok("PATCH", "/admin/accounts/milo/quotas", { storage_bytes: used }, T.sully);
2449+ assert.deepEqual(set.storage_bytes, { used, limit: used });
2450+ const full = new RegExp(`milo has reached its storage limit of ${used} bytes`);
2451+ commit("b.txt");
2452+ const refused = push("HEAD:main");
2453+ assert.notEqual(refused.status, 0);
2454+ assert.match(refused.stderr, new RegExp(`remote: Push refused: ${full.source}`));
2455+ assert.match(refused.stderr, /storage limit reached/);
2456+ const lfsOid = (bytes) => createHash("sha256").update(bytes).digest("hex");
2457+ const lfsBlob = randomBytes(1000);
2458+ const batch = async () => {
2459+ const r = await fetch(`${base}/milo/hoard.git/info/lfs/objects/batch`, { method: "POST", headers: { authorization: `Bearer ${T.milo}`, accept: "application/vnd.git-lfs+json", "content-type": "application/vnd.git-lfs+json" }, body: JSON.stringify({ operation: "upload", objects: [{ oid: lfsOid(lfsBlob), size: lfsBlob.length }] }) });
2460+ assert.equal(r.status, 200);
2461+ return (await r.json()).objects[0];
2462+ };
2463+ const lfs = await batch();
2464+ assert.equal(lfs.error.code, 507);
2465+ assert.match(lfs.error.message, full);
2466+ const put = await fetch(`${base}/milo/hoard.git/info/lfs/objects/${lfsOid(lfsBlob)}`, { method: "PUT", headers: { authorization: `Bearer ${T.milo}`, "content-type": "application/octet-stream" }, body: lfsBlob });
2467+ assert.equal(put.status, 507);
2468+ const late = await upload(randomBytes(10));
2469+ assert.equal(late.status, 403);
2470+ assert.match((await late.json()).message, full);
2471+ assert.match((await status(403, "PUT", `${R}/contents/c.txt`, { message: "c", content: b64("more") }, T.milo)).message, full);
2472+ assert.match((await status(403, "POST", `${R}/git/blobs`, { content: "more" }, T.milo)).message, full);
2473+
2474+ // ------------------------------------------------------------ reads, ref deletions and ref moves still work
2475+ assert.equal((await ok("GET", `${R}/contents/a.txt`, undefined, T.milo)).size, 4000);
2476+ assert.equal(push(":refs/heads/topic").status, 0);
2477+ assert.equal(push("HEAD~1:refs/heads/copy").status, 0);
2478+ assert.deepEqual((await ok("GET", `${R}/branches`, undefined, T.milo)).map((b) => b.name).sort(), ["copy", "main"]);
2479+
2480+ // ------------------------------------------------------------ deleting brings usage down, and there is room again
2481+ await status(204, "DELETE", `${R}/releases/assets/${asset.id}`, undefined, T.milo);
2482+ await until("the asset's bytes to be given back", async () => (await usage()).used === used - 3000);
2483+ assert.ok((await batch()).actions.upload);
2484+ await ok("PATCH", "/admin/accounts/milo/quotas", { storage_bytes: null }, T.sully);
2485+ assert.equal(push("HEAD:main").status, 0);
2486+});
2487+
24112488 /** Set rita's repository quota (site administrators only). */
24122489 async function setRitaRepos(n) {
24132490 await ok("PATCH", "/admin/accounts/rita/quotas", { repositories: n }, T.sully);
modifiedapps/diffinery/test/unit/quotas.test.ts+72−2View file
@@ -1,5 +1,5 @@
11 import { describe, expect, it } from "vitest";
2-import { checkRepositoryQuota, DEFAULT_QUOTAS, endCompute, mayStart, monthStart, quotasFor, setQuotas, startCompute, usageOf } from "../../src/lib/quotas.js";
2+import { checkRepositoryQuota, checkStorageQuota, DEFAULT_QUOTAS, endCompute, mayStart, monthStart, quotasFor, setQuotas, startCompute, storageRefusal, storedBytes, usageOf } from "../../src/lib/quotas.js";
33 import type { Env } from "../../src/env.js";
44 import { testD1 } from "./stubs/d1.js";
55
@@ -51,7 +51,7 @@
5151 usage("j4", "acme", "job", "2026-09-10T10:00:00.000Z", "2026-09-10T11:00:00.000Z"); // last month
5252 usage("j5", "acme", "job", "2026-10-12T00:00:00.000Z", null); // lost its end: counted as ended after 7 hours
5353 usage("j6", "ann", "job", "2026-10-13T11:00:00.000Z", null); // someone else's
54- expect(await usageOf(env, "acme", now)).toEqual({ hosted_minutes: 11 + 30 + 60 + 20 + 420, hosted_jobs: 1, sandboxes: 1, repositories: 0 });
54+ expect(await usageOf(env, "acme", now)).toEqual({ hosted_minutes: 11 + 30 + 60 + 20 + 420, hosted_jobs: 1, sandboxes: 1, repositories: 0, storage_bytes: 0 });
5555 expect(monthStart(now)).toBe("2026-10-01T00:00:00.000Z");
5656 });
5757
@@ -85,3 +85,73 @@
8585 await checkRepositoryQuota(env, "ann", "ann");
8686 });
8787 });
88+
89+describe("storage quota", () => {
90+ /** acme's repositories r1 (reported 1000 bytes) and r2 (never measured), ann's r3; `measured` lists the repositories asked to report. */
91+ function storage(vars: Record<string, string> = {}) {
92+ const s = setup(vars);
93+ const repo = (id: string, owner: string) => s.DB.sqlite.prepare("INSERT INTO repos (id, owner_id, name, description, visibility, default_branch, created_at, updated_at) VALUES (?, ?, ?, '', 'private', 'main', ?, ?)").run(id, owner, id, at, at);
94+ const report = (id: string, bytes: number) => s.DB.sqlite.prepare("INSERT INTO repo_storage (repo_id, bytes, updated_at) VALUES (?, ?, ?) ON CONFLICT(repo_id) DO UPDATE SET bytes = excluded.bytes").run(id, bytes, at);
95+ repo("r1", "acme");
96+ repo("r2", "acme");
97+ repo("r3", "ann");
98+ report("r1", 1000);
99+ report("r3", 5000);
100+ const measured: string[] = [];
101+ // A repository asked to report writes its row, as RepoDO.reportStorage does.
102+ const REPO = { idFromName: (id: string) => id, get: (id: string) => ({ reportStorage: async () => (measured.push(id), report(id, 200)) }) };
103+ const env = { ...s.env, REPO } as unknown as Env;
104+ return { ...s, env, repo, report, measured };
105+ }
106+
107+ it("is 10 GB by default, from QUOTA_STORAGE_BYTES, or a site administrator's override up to 1 PiB", async () => {
108+ expect(DEFAULT_QUOTAS.storage_bytes).toBe(10 * 1024 ** 3);
109+ const { env } = storage();
110+ expect((await quotasFor(env, "acme")).storage_bytes).toBe(10 * 1024 ** 3);
111+ expect((await quotasFor(storage({ QUOTA_STORAGE_BYTES: "2048" }).env, "acme")).storage_bytes).toBe(2048);
112+ await setQuotas(env, "acme", { storage_bytes: 100 * 1024 ** 4 }, "root");
113+ expect((await quotasFor(env, "acme")).storage_bytes).toBe(100 * 1024 ** 4);
114+ await expect(setQuotas(env, "acme", { storage_bytes: 2 ** 51 }, "root")).rejects.toThrow(/whole number/);
115+ expect((await quotasFor(env, "root")).storage_bytes).toBe(-1);
116+ });
117+
118+ it("adds up the account's repositories, measuring those that never reported, and its packages", async () => {
119+ const { env, DB, measured } = storage();
120+ // r2 never reported: it is asked once, then its report counts.
121+ expect(await storedBytes(env, "acme")).toBe(1000 + 200);
122+ expect(measured).toEqual(["r2"]);
123+ expect(await storedBytes(env, "acme")).toBe(1200);
124+ expect(measured).toEqual(["r2"]);
125+ // A repository counting its own bytes replaces its report.
126+ expect(await storedBytes(env, "acme", { repoId: "r1", bytes: 1500 })).toBe(1500 + 200);
127+ // Packages: each blob once however many of the account's packages use it, plus the manifests.
128+ const p = (id: string, owner: string) => DB.sqlite.prepare("INSERT INTO packages (id, owner_id, name, created_at, updated_at) VALUES (?, ?, ?, ?, ?)").run(id, owner, id, at, at);
129+ p("p1", "acme");
130+ p("p2", "acme");
131+ p("p3", "ann");
132+ DB.sqlite.prepare("INSERT INTO registry_blobs (digest, size, r2_key, created_at) VALUES ('sha256:a', 300, 'k1', ?), ('sha256:b', 40, 'k2', ?)").run(at, at);
133+ DB.sqlite.prepare("INSERT INTO package_blobs (package_id, digest, created_at) VALUES ('p1', 'sha256:a', ?), ('p2', 'sha256:a', ?), ('p3', 'sha256:b', ?)").run(at, at, at);
134+ DB.sqlite.prepare("INSERT INTO package_manifests (package_id, digest, media_type, size, created_at) VALUES ('p1', 'sha256:m', 'x', 7, ?)").run(at);
135+ expect(await storedBytes(env, "acme")).toBe(1200 + 300 + 7);
136+ expect(await storedBytes(env, "ann")).toBe(5000 + 40);
137+ expect((await usageOf(env, "acme")).storage_bytes).toBe(1507);
138+ // A deleted repository's report goes with it.
139+ DB.sqlite.prepare("DELETE FROM repos WHERE id = 'r1'").run();
140+ expect(await storedBytes(env, "acme")).toBe(507);
141+ });
142+
143+ it("refuses a write that would take the account past its quota, naming the account and the limit", async () => {
144+ const { env } = storage();
145+ await setQuotas(env, "acme", { storage_bytes: 1300 }, "root");
146+ expect(await storageRefusal(env, "acme", "acme", 100)).toBeNull();
147+ expect(await storageRefusal(env, "acme", "acme", 101)).toMatch(/^acme has reached its storage limit of 1300 bytes/);
148+ await expect(checkStorageQuota(env, "acme", "acme", 101)).rejects.toThrow(/403: acme has reached its storage limit of 1300 bytes/);
149+ // The repository's own count, fresher than its report, decides.
150+ expect(await storageRefusal(env, "acme", "acme", 0, { repoId: "r1", bytes: 1200 })).toMatch(/storage limit/);
151+ await setQuotas(env, "acme", { storage_bytes: -1 }, "root");
152+ expect(await storageRefusal(env, "acme", "acme", 1e12)).toBeNull();
153+ // Accounts run by a site administrator have no limit.
154+ await setQuotas(env, "ops", { storage_bytes: 0 }, "root");
155+ expect(await storageRefusal(env, "ops", "ops", 1e12)).toBeNull();
156+ });
157+});
addedapps/diffinery/test/unit/storage.test.ts+66−0View file
@@ -1,0 +1,66 @@
1+import { describe, expect, it } from "vitest";
2+import { DatabaseSync } from "node:sqlite";
3+import { randomBytes } from "node:crypto";
4+import { deflate, hashObject, utf8 } from "@diffinery/git-core";
5+import { RepoObjectStore } from "../../src/repo/objects.js";
6+import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
7+
8+/** A minimal SqlStorage over node:sqlite with the repository schema (RETURNING gives rows back too). */
9+function sqlStorage(): SqlStorage {
10+ const db = new DatabaseSync(":memory:");
11+ db.exec(REPO_SCHEMA);
12+ const sql = {
13+ exec(query: string, ...bindings: unknown[]) {
14+ const stmt = db.prepare(query);
15+ const rows = /^\s*select|returning/i.test(query) ? stmt.all(...(bindings as never[])) : (stmt.run(...(bindings as never[])), []);
16+ return { toArray: () => rows, one: () => rows[0] } as unknown as SqlStorageCursor<Record<string, SqlStorageValue>>;
17+ },
18+ } as unknown as SqlStorage;
19+ migrateRepoSchema(sql);
20+ return sql;
21+}
22+
23+function setup() {
24+ const sql = sqlStorage();
25+ const r2 = new Map<string, Uint8Array>();
26+ const bucket = { put: async (key: string, body: Uint8Array) => void r2.set(key, body) } as unknown as R2Bucket;
27+ let added = 0;
28+ const store = new RepoObjectStore(sql, bucket, "r1", () => added++);
29+ return { sql, r2, store, added: () => added };
30+}
31+
32+describe("git object storage accounting", () => {
33+ it("keeps a running total of the compressed bytes stored, counting each object once", async () => {
34+ const { store, r2, added } = setup();
35+ expect(store.sizeBytes()).toBe(0);
36+ const text = utf8("hello\n".repeat(1000));
37+ await store.write("blob", text);
38+ await store.write("blob", text);
39+ expect(store.sizeBytes()).toBe(deflate(text).length);
40+ expect(added()).toBe(1);
41+ // Large objects go to R2 and count what R2 keeps.
42+ const big = new Uint8Array(randomBytes(700 * 1024));
43+ await store.write("blob", big);
44+ expect(r2.size).toBe(1);
45+ expect(store.sizeBytes()).toBe(deflate(text).length + deflate(big).length);
46+ // A pushed object stored from its zlib data counts that.
47+ const small = utf8("pushed\n");
48+ await store.writeDeflated(await hashObject("blob", small), "blob", small.length, deflate(small));
49+ expect(store.sizeBytes()).toBe(deflate(text).length + deflate(big).length + deflate(small).length);
50+ expect(added()).toBe(3);
51+ });
52+
53+ it("adds up a repository from before the total was kept, and again after objects were deleted", async () => {
54+ const { sql, store } = setup();
55+ // Rows as older code wrote them: no stored size; an R2 object counts its inflated size.
56+ sql.exec("INSERT INTO objects (oid, type, size, data, r2) VALUES ('a', 3, 10, ?, 0)", new Uint8Array(4));
57+ sql.exec("INSERT INTO objects (oid, type, size, data, r2) VALUES ('b', 3, 900000, NULL, 1)");
58+ expect(store.sizeBytes()).toBe(4 + 900000);
59+ await store.write("blob", utf8("new\n"));
60+ expect(store.sizeBytes()).toBe(4 + 900000 + deflate(utf8("new\n")).length);
61+ // A purge deletes objects, then has the total added up again.
62+ sql.exec("DELETE FROM objects WHERE oid = 'b'");
63+ store.recount();
64+ expect(store.sizeBytes()).toBe(4 + deflate(utf8("new\n")).length);
65+ });
66+});
modifiedcontainers/runner/src/import-plan.ts+5−0View file
@@ -64,6 +64,11 @@
6464 return /exceeds \d+ ?MB|split it into smaller pushes|HTTP 413|Request Entity Too Large|RPC failed|remote end hung up|HTTP 50[023]|unexpected disconnect|pack exceeds/i.test(output);
6565 }
6666
67+/** The server's reason when the push was refused because the owner reached its storage limit, or null. */
68+export function storageRejection(output: string): string | null {
69+ return /^remote: Push refused: (.* has reached its storage limit .*?)\s*$/m.exec(output)?.[1] ?? null;
70+}
71+
6772 /** The push was refused by push protection. */
6873 export function isSecretRejection(output: string): boolean {
6974 return /secret detected|Push blocked: possible secrets/i.test(output);
modifiedcontainers/runner/src/importer.ts+4−1View file
@@ -19,7 +19,7 @@
1919 import path from "node:path";
2020 import type { ImportGitResult, ImportProgress, ImportSecretFinding, ImportSpec } from "@diffinery/shared";
2121 import { RunnerApi } from "./api.js";
22-import { IMPORT_REF_PREFIX, batchBySize, furthestWithin, importDestination, importRefKind, isSecretRejection, isSizeRejection, parsePorcelain, parseSecretFindings, sortRefsForImport } from "./import-plan.js";
22+import { IMPORT_REF_PREFIX, batchBySize, furthestWithin, importDestination, importRefKind, isSecretRejection, isSizeRejection, parsePorcelain, parseSecretFindings, sortRefsForImport, storageRejection } from "./import-plan.js";
2323 import { exec } from "./util.js";
2424
2525 export interface ImportAgentOptions {
@@ -432,6 +432,9 @@
432432 found,
433433 );
434434 }
435+ // Past the owner's storage quota nothing more can be stored: say so rather than "push failed".
436+ const full = storageRejection(text);
437+ if (full) throw new ImportError(full);
435438 if (isSizeRejection(text)) {
436439 if (this.budget / 2 >= MIN_BUDGET) {
437440 this.budget = Math.floor(this.budget / 2);
modifiedcontainers/runner/test/import.test.ts+3−1View file
@@ -5,7 +5,7 @@
55 import path from "node:path";
66 import { describe, expect, it } from "vitest";
77 import type { ImportSpec } from "@diffinery/shared";
8-import { batchBySize, furthestWithin, importDestination, importRefKind, isSecretRejection, isSizeRejection, parsePorcelain, parseSecretFindings, sortRefsForImport } from "../src/import-plan.js";
8+import { batchBySize, furthestWithin, importDestination, importRefKind, isSecretRejection, isSizeRejection, parsePorcelain, parseSecretFindings, sortRefsForImport, storageRejection } from "../src/import-plan.js";
99 import { pushFailure, transfer } from "../src/importer.js";
1010
1111 describe("import chunk planning", () => {
@@ -50,6 +50,8 @@
5050 expect(isSizeRejection("error: RPC failed; HTTP 413 curl 22")).toBe(true);
5151 expect(isSizeRejection("! [rejected] main (non-fast-forward)")).toBe(false);
5252 expect(isSizeRejection("error: remote unpack failed: the push needs more memory than one push may use; split it into smaller pushes (Array buffer allocation failed)")).toBe(true);
53+ expect(storageRejection("remote: Push refused: acme has reached its storage limit of 1000 bytes (git objects); delete some \nerror: failed to push some refs")).toBe("acme has reached its storage limit of 1000 bytes (git objects); delete some");
54+ expect(storageRejection("error: failed to push some refs")).toBeNull();
5355 expect(parsePorcelain("To x\n*\trefs/heads/a:refs/heads/a\t[new branch]\n!\trefs/heads/b:refs/heads/b\t[remote rejected] (invalid ref name)\nDone\n")).toEqual([
5456 { ref: "refs/heads/a", ok: true, reason: "[new branch]" },
5557 { ref: "refs/heads/b", ok: false, reason: "invalid ref name" },